WO2012128070A1 - 画像処理装置および画像処理方法 - Google Patents
画像処理装置および画像処理方法 Download PDFInfo
- Publication number
- WO2012128070A1 WO2012128070A1 PCT/JP2012/056084 JP2012056084W WO2012128070A1 WO 2012128070 A1 WO2012128070 A1 WO 2012128070A1 JP 2012056084 W JP2012056084 W JP 2012056084W WO 2012128070 A1 WO2012128070 A1 WO 2012128070A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- viewpoint
- unit
- color image
- depth
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims abstract description 184
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000000034 method Methods 0.000 claims abstract description 130
- 230000008569 process Effects 0.000 claims description 89
- 230000005540 biological transmission Effects 0.000 claims description 58
- 238000003384 imaging method Methods 0.000 claims description 28
- 230000010365 information processing Effects 0.000 claims 1
- 238000003702 image correction Methods 0.000 description 27
- 238000005516 engineering process Methods 0.000 description 26
- 238000012937 correction Methods 0.000 description 25
- 238000010586 diagram Methods 0.000 description 25
- 238000004891 communication Methods 0.000 description 18
- 238000010606 normalization Methods 0.000 description 18
- 239000000284 extract Substances 0.000 description 15
- 230000006870 function Effects 0.000 description 14
- 230000003287 optical effect Effects 0.000 description 12
- 230000015572 biosynthetic process Effects 0.000 description 9
- 238000006243 chemical reaction Methods 0.000 description 9
- 230000005236 sound signal Effects 0.000 description 9
- 238000003786 synthesis reaction Methods 0.000 description 9
- 239000011521 glass Substances 0.000 description 8
- 239000000203 mixture Substances 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 5
- 230000003321 amplification Effects 0.000 description 4
- 238000003199 nucleic acid amplification method Methods 0.000 description 4
- 239000002131 composite material Substances 0.000 description 3
- 239000004065 semiconductor Substances 0.000 description 3
- 238000007667 floating Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 210000003127 knee Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
- G06T15/205—Image-based rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/18—Image warping, e.g. rearranging pixels individually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/15—Processing image signals for colour aspects of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
- H04N13/178—Metadata, e.g. disparity information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
- H04N13/31—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays using parallax barriers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/349—Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0077—Colour aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Definitions
- the present technology relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method capable of generating a color image of a display viewpoint using a color image and a depth image of a predetermined viewpoint. .
- 3D images have attracted attention.
- the shutter for the left eye is opened when displaying one of the images of two viewpoints, and the shutter for the right eye is worn when displaying the other image.
- a method of viewing an image of two viewpoints (hereinafter referred to as a method with glasses) is common.
- a viewing method capable of viewing 3D images without wearing glasses is increasing.
- images of three or more viewpoints are displayed so that the viewable angle differs for each viewpoint, and the viewer views the images of any two viewpoints with the left and right eyes, respectively.
- 3D images can be viewed without wearing
- a color image and a depth image of a predetermined viewpoint are acquired, and using the color image and the depth image, multi-view color as a display viewpoint including a viewpoint other than the predetermined viewpoint.
- Methods have been devised to generate and display images.
- a multi-viewpoint is a viewpoint of three or more viewpoints.
- Patent Document 1 As a method of encoding a multi-view color image and a depth image, a method of separately encoding a color image and a depth image has been proposed (see, for example, Patent Document 1).
- the present technology has been made in view of such a situation, and is configured to be able to generate a color image of a display viewpoint using a color image of a predetermined viewpoint and a parallax related image.
- the image processing apparatus encodes a color image of a viewpoint and a depth image of the viewpoint to generate a bitstream, and performs warping using the color image and the depth image.
- a generation unit configured to generate viewpoint generation information used when generating a color image of the display viewpoint according to a method of generating a color image of the display viewpoint obtained by performing the processing; It is an image processing apparatus provided with the transmission part which transmits a bit stream and the said information for viewpoint generation produced
- the image processing method according to the first aspect of the present technology corresponds to the image processing device according to the first aspect of the present technology.
- a color image of a viewpoint and a depth image of the viewpoint are encoded to generate a bit stream, and are obtained by performing warping processing using the color image and the depth image.
- viewpoint generation information used when generating a color image of the display viewpoint is generated, and the bit stream and the viewpoint generation information are transmitted.
- the image processing apparatus performs warping processing using a color image of a viewpoint, a bit stream obtained as a result of encoding a depth image of the viewpoint, the color image, and the depth image.
- a receiver for receiving information for viewpoint generation used in generating a color image of the display viewpoint which is generated according to a method for generating a color image of the display viewpoint obtained by the method; and the bit stream received by the receiver Using the color image and the depth image, the color image and the depth image generated by the decoding unit, and the viewpoint generation information received by the reception unit.
- An image processing apparatus including a generation unit that generates a color image of the display viewpoint by performing a warping process; That.
- the image processing method according to the second aspect of the present technology corresponds to the image processing device according to the second aspect of the present technology.
- the second aspect of the present technology it is obtained by performing warping processing using a color image of a viewpoint, a bit stream obtained as a result of encoding a depth image of the viewpoint, and the color image and the depth image.
- Viewpoint generation information used to generate a color image of the display viewpoint which is generated according to a method of generating a color image of the display viewpoint, is received, the received bit stream is decoded, and the color image and the color image are A depth image is generated, and a warping process is performed using the color image, the depth image, and the viewpoint generation information to generate a color image of the display viewpoint.
- the image processing apparatus can be realized by causing a computer to execute a program.
- a program to be executed by a computer can be provided by transmitting via a transmission medium or recording on a recording medium .
- the first aspect of the present technology it is possible to transmit information necessary to generate a color image of a display viewpoint using a color image and a depth image of a predetermined viewpoint.
- a color image of a display viewpoint can be generated using a color image and a depth image of a predetermined viewpoint.
- FIG. 1 It is a figure which shows the example of the syntax of PPS of FIG. It is a figure which shows the example of the syntax of a slice header. It is a figure which shows the example of the syntax of a slice header. It is a flowchart explaining the encoding process of the encoding apparatus of FIG. It is a flowchart explaining the multiview coding process of FIG. It is a block diagram showing an example of composition of a 2nd embodiment of a decoding device as an image processing device to which this art is applied. It is a block diagram which shows the structural example of the multi-viewpoint image decoding part of FIG. It is a flowchart explaining the multiview decoding process of the multiview image decoding part of FIG. It is a figure explaining parallax and depth.
- FIG. 1 is a diagram showing an example of a configuration of an embodiment of a computer.
- FIG. 1 is a diagram illustrating a schematic configuration example of a television device to which the present technology is applied. It is a figure which shows the example of a schematic structure of the mobile telephone to which this technique is applied. It is a figure showing an example of outline composition of a recording and reproducing device to which this art is applied. It is a figure showing an example of outline composition of an imaging device to which this art is applied.
- FIG. 25 is a diagram for explaining parallax and depth.
- the depth of the subject M from the camera c1 (camera c2)
- the depth Z which is the distance of the direction, is defined by the following equation (a).
- L is the distance between the position C1 and the position C2 in the horizontal direction (hereinafter referred to as the inter-camera distance).
- d is the position of the subject M on the color image taken with the camera c2 from the distance u1 in the horizontal direction from the center of the color image of the position of the subject M on the color image taken with the camera c1 A value obtained by subtracting the horizontal distance u2 from the center of the color image, that is, the parallax.
- f is the focal length of the camera c1, and in equation (a), the focal lengths of the camera c1 and the camera c2 are the same.
- a parallax image representing the parallax d of a color image of two viewpoints photographed by the camera c1 and the camera c2 and a depth image representing the depth Z are collectively referred to as a depth image (parallax related image).
- the depth image may be an image representing the parallax d or the depth Z, and the pixel value of the depth image (parallax related image) is not the parallax d or the depth Z itself, but the parallax d is normal. It is possible to adopt a normalized value, a value obtained by normalizing the reciprocal 1 / Z of the depth Z, or the like.
- a value I obtained by normalizing the parallax d with 8 bits (0 to 255) can be obtained by the following equation (b).
- the normalization bit number of the parallax d is not limited to 8 bits, It is also possible to set it as another bit number, such as 10 bits and 12 bits.
- D max is the maximum value of disparity d
- D min is the minimum value of disparity d.
- the maximum value D max and the minimum value D min may be set in units of one screen or may be set in units of plural screens.
- a value y obtained by normalizing the reciprocal 1 / Z of the depth Z with 8 bits (0 to 255) can be obtained by the following equation (c).
- the normalized bit number of the reciprocal 1 / Z of the depth Z is not limited to 8 bits, and may be another bit number such as 10 bits or 12 bits.
- Z far is the maximum value of depth Z
- Z near is the minimum value of depth Z.
- the maximum value Z far and the minimum value Z near may be set in units of one screen or may be set in units of plural screens.
- a parallax image having a pixel value as a value I obtained by normalizing the parallax d and an inverse number 1 of the depth Z A depth image (parallax related image) is generically referred to as a depth image in which a pixel value is a value y obtained by normalizing / Z.
- the color format of the depth image (parallax related image) is assumed to be YUV 420 or YUV 400, but other color formats can also be used.
- the value I or the value y is taken as depth information. Further, the mapping of the value I or the value y is used as a depth map.
- FIG. 2 is a block diagram showing a configuration example of a first embodiment of an encoding device as an image processing device to which the present technology is applied.
- the encoding device 10 of FIG. 2 includes a multi-view color image capturing unit 11, a multi-view color image correction unit 12, a multi-view parallax related image generation unit 13, a viewpoint generation information generation unit 14, and a multi-view image coding unit 15. It consists of
- the encoding apparatus 10 encodes a color image of a predetermined viewpoint and a parallax related image, and uses information of the color image of the predetermined viewpoint and the parallax related image to generate information necessary to generate a color image of a viewpoint other than the viewpoint.
- the information for viewpoint generation which is is added and transmitted.
- the multi-view color image capturing unit 11 of the encoding device 10 captures a multi-view color image and supplies the multi-view color image correction unit 12 as a multi-view color image. Further, the multi-view color image capturing unit 11 generates imaging information such as the number of viewpoints of a color image, external parameters, range information and the like as imaging information, and supplies the information to the viewpoint generation information generating unit 14.
- the external parameters are parameters that define the horizontal position of the multi-viewpoint color image capturing unit 11.
- the range information is the world coordinate value of the position in the depth direction that can be taken in the multiview parallax related image (multiview depth image) when the parallax related image generated by the multiview parallax related image generation unit 13 is a depth image. made from the minimum value (Min. Z near) the maximum value (maximum value Z _far).
- the minimum value and the maximum value will be referred to as the depth minimum and the depth maximum, respectively.
- the range information is the minimum value (minimum value D min ) of parallax on world coordinates that can be taken in the multi-view parallax related image.
- a maximum value (maximum value D max ) and information for specifying a color image of a viewpoint based on which a parallax value is obtained.
- the minimum value and the maximum value are hereinafter referred to as a parallax minimum value and a parallax maximum value, respectively.
- the multi-viewpoint color image correction unit 12 performs color correction, luminance correction, distortion correction, and the like on the multi-viewpoint color image supplied from the multi-viewpoint color image pickup unit 11. As a result, the focal length in the horizontal direction (X direction) of the multi-view color image capturing unit 11 in the corrected multi-view color image becomes common to all the views.
- the multi-view color image correction unit 12 supplies the corrected multi-view color image as a multi-view correction color image to the multi-view parallax related image generation unit 13 and the multi-view image coding unit 15. Further, the multi-viewpoint color image correction unit 12 generates information on the multi-viewpoint corrected color image such as internal parameters as color image information, and supplies the information to the viewpoint generation information generation unit 14.
- the internal parameters are the focal length in the horizontal direction of the multi-view color image capturing unit 11 common to all the viewpoints in the multi-view corrected color image, and the main point that is the image center, that is, the horizontal direction And the position of Note that the horizontal position of the principal point may be different for each viewpoint.
- the multiview parallax related image generation unit 13 generates a multiview parallax related image from the multiview corrected color image supplied from the multiview color image correction unit 12. Then, the multi-viewpoint parallax related image generation unit 13 supplies the generated multi-viewpoint parallax related image to the multi-viewpoint image encoding unit 15 as a multi-viewpoint parallax related image.
- the multi-viewpoint parallax related image generation unit 13 is image type information (depth), which is information indicating whether the number of viewpoints of the multi-viewpoint parallax related image or the multi-viewpoint parallax related image is a depth image or a parallax image.
- image type information information indicating whether the number of viewpoints of the multi-viewpoint parallax related image or the multi-viewpoint parallax related image is a depth image or a parallax image.
- Information on multi-viewpoint parallax related images such as image identification information is generated as parallax related image information (depth image information). Then, the multi-viewpoint parallax related image generation unit 13 supplies parallax related image information to the viewpoint generation information generation unit 14.
- the viewpoint generation information generation unit 14 functions as a generation unit, and generates viewpoint generation information according to a predetermined method of generating color images of other viewpoints using the multi-viewpoint corrected color image and the multi-viewpoint parallax related image.
- the viewpoint generation information generation unit 14 includes the number of viewpoints of the color image supplied from the multiview color image capturing unit 11 and the number of viewpoints of the parallax related images supplied from the multiview parallax related image generation unit 13.
- To generate color image specification information and parallax related image specification information depth image specification information).
- the color image specification information is information that specifies a color image
- the parallax related image specification information is information that specifies a parallax related image.
- the viewpoint generation information generating unit 14 sets external parameters of the viewpoints for each viewpoint corresponding to the multi-view parallax related image. Generates an external parameter flag that indicates presence or absence. Furthermore, based on the internal parameters included in the color image information from the multi-view color image correction unit 12, the viewpoint generation information generation unit 14 generates the internal parameters of the viewpoint for each of the viewpoints corresponding to the multi-view parallax related image.
- the viewpoint generation information generation unit 14 Generate an internal parameter flag that indicates the presence or absence of Then, the viewpoint generation information generation unit 14 generates viewpoint generation information from the color image specification information, the parallax related image specification information, the external parameter flag, the internal parameter flag, the imaging information, the color image information, and the parallax related image information.
- the viewpoint generation information includes color image specification information, parallax related image specification information, external parameter information, internal parameter information, number of color image viewpoints, range information, number of parallax related image viewpoints, and image type information. .
- the external parameter information includes an external parameter flag and an external parameter of the viewpoint corresponding to the multiview parallax related image
- the internal parameter information includes an internal parameter flag and an internal parameter of the viewpoint corresponding to the multiview parallax related image It consists of
- the viewpoint generation information generation unit 14 supplies the generated viewpoint generation information to the multi-viewpoint image coding unit 15.
- the multi-view image encoding unit 15 functions as an encoding unit, and performs the multi-view corrected color image from the multi-view color image correction unit 12 and the multi-view parallax related image from the multi-view parallax related image generation unit 13 to MVC (Multiview Coding is performed by the same method as the Video Coding method, and the viewpoint generation information from the viewpoint generation information generation unit 14 is added.
- the multi-viewpoint image encoding unit 15 functions as a transmission unit, and transmits the resultant bit stream as a coded bit stream.
- FIG. 3 is a diagram showing a configuration example of an access unit of a coded bit stream generated by the coding device 10 of FIG.
- the access unit of the coded bit stream is composed of an SPS (Sequence Parameter Set), a Subset SPS, a PPS (Picture Parameter Set), an SEI, and a slice (Slice).
- the number of viewpoints of the multi-viewpoint corrected color image and the multi-viewpoint parallax related image is two. Then, the color image A after correction of one of the two-viewpoint corrected color images is encoded as a base view. Also, the color image B after correction of the other viewpoint, the parallax related image A corresponding to the color image A, and the parallax related image B corresponding to the color image B are encoded as a non-base view.
- the arrangement order of slices is, for example, from the top, a slice of color image A encoded as a base view, a slice of disparity related image A encoded as a non-base view, a slice of color image B, disparity related It becomes the order of the slice of image B.
- information for specifying PPS is described.
- the SPS is a header that contains information on base view encoding.
- Subset SPS is an extension header that contains information on base view and non-base view encoding.
- the PPS is a header including information indicating the coding mode of the entire picture, SPS, and information for specifying the Subset SPS.
- SEI is additional information that is not essential for decoding, and includes information for viewpoint generation and the like.
- the PPS When decoding a color image A encoded as a base view, the PPS is referred to based on the information for specifying the PPS described in the header portion of the color image A, and the SPS described in the PPS is specified.
- the SPS is referenced based on the information to
- the PPS when decoding the parallax related image A encoded as the non-base view, the PPS is referred to based on the information for specifying the PPS described in the header of the parallax related image A. Also, the Sub SPS is referred to based on the information for specifying the Sub SPS described in the PPS. Also at the time of decoding of the color image B and the parallax related image B encoded as the non-base view, the PPS is referred to and the Sub SPS is referred to similarly to the decoding of the parallax related image A.
- FIG. 4 is a view showing a description example of a part of SEI.
- the number of viewpoints (num_color_view_minus_1) of a color image is described on the second line from the top on the left side of the SEI in FIG. 4, and the number of viewpoints (num_depth_view_minus_1) of a parallax related image is described on the third line.
- the view ID (color_view_id) of the color image is described as color image specification information of the color image of each viewpoint
- the parallax related image of each viewpoint The view ID (depth_view_id) of the parallax-related image is described as the parallax-related image specifying information.
- an internal parameter flag of the internal parameter information is described for each viewpoint corresponding to the multi-viewpoint parallax related image.
- an external parameter flag in the external parameter information is described for each viewpoint corresponding to the multi-view parallax related image.
- the real number x is described as a floating point number using the prec value, the sign value, the exponent value, and the mantissa value defined by the following equation (1).
- prec is a prec value
- s is a sign value
- e is an exponent value
- n is a mantissa value. Therefore, the sign value, the exponent value and the mantissa value represent the sign, exponent and mantissa of the real number x, respectively.
- prec_focal_length the prec value of the focal distance in the horizontal direction is shown on the 11th and 13th to 15th lines from the top on the left side of FIG.
- sign value sign_focal_length_x
- exponent_focal_length_x an exponent value
- mantissa_focal_length_x mantissa value
- prec_principal_point a value common to each viewpoint corresponding to the multi-viewpoint parallax related image is described as a prec value (prec_principal_point) of the horizontal position of the principal point. Also, in the 17th to 19th lines from the top of the left side of FIG. It is described for each viewpoint corresponding to the image.
- prec_translation_param a value common to each viewpoint corresponding to the multi-view parallax related image is described as a prec value (prec_translation_param) of the external parameter.
- prec_translation_param a value common to each viewpoint corresponding to the multi-view parallax related image.
- the external parameter sign values (sign_translation_x), exponent values (exponent_translation_x), and mantissa values (mantissa_translation_x) are the viewpoints corresponding to the multiview parallax related image, respectively. Described for each
- image type information (depth_map_flag) is described.
- the minimum depth value common to each viewpoint corresponding to the multi-viewpoint parallax related image in the range information (depth_nearest) and depth maximum value (depth_farthest) are described respectively.
- range information in the case where the image type information indicates that the image is a parallax image is described.
- the disparity minimum value (disparity_minimum) and the disparity maximum value (disparity_maximum) are described in the 16th and 17th lines, respectively.
- the parallax minimum value and the parallax maximum value differ depending on the viewpoint, they are generated and described for each viewpoint.
- the view ID (reference_depth_view) of the color image is for each viewpoint corresponding to the multiview parallax related image. Described in
- FIG. 5 is a flowchart illustrating the encoding process of the encoding device 10 of FIG.
- step S10 of FIG. 5 the multi-view color image capturing unit 11 of the encoding device 10 captures a multi-view color image and supplies the multi-view color image correction unit 12 as a multi-view color image.
- step S ⁇ b> 11 the multi-view color image capturing unit 11 generates imaging information and supplies the imaging information to the viewpoint generation information generating unit 14.
- step S12 the multi-view color image correction unit 12 performs color correction, luminance correction, distortion correction, and the like on the multi-view color image supplied from the multi-view color image capturing unit 11.
- the multi-view color image correction unit 12 supplies the corrected multi-view color image as a multi-view correction color image to the multi-view parallax related image generation unit 13 and the multi-view image coding unit 15.
- step S13 the multi-view color image correction unit 12 generates color image information and supplies the color image information to the viewpoint generation information generation unit 14.
- step S14 the multiview parallax related image generation unit 13 generates a multiview parallax related image from the multiview corrected color image supplied from the multiview color image correction unit 12. Then, the multi-viewpoint parallax related image generation unit 13 supplies the generated multi-viewpoint parallax related image to the multi-viewpoint image encoding unit 15 as a multi-viewpoint parallax related image.
- step S ⁇ b> 15 the multi-viewpoint parallax related image generating unit 13 generates parallax related image information and supplies the generated information to the viewpoint generation information generating unit 14.
- step S16 the viewpoint generation information generation unit 14 generates color image specification information, parallax related image specification information, an internal parameter flag, and an external parameter flag based on imaging information, color image information, and parallax related image information. Do.
- the viewpoint generation information generation unit 14 generates viewpoint generation information.
- the viewpoint generation information generation unit 14 includes color image specification information, parallax related image specification information, the number of color image viewpoints, external parameter information, internal parameter information, range information, the number of parallax related image viewpoints, Image type information is generated as viewpoint generation information.
- the viewpoint generation information generation unit 14 supplies the generated viewpoint generation information to the multi-viewpoint image coding unit 15.
- step S18 the multi-viewpoint image encoding unit 15 encodes a multi-viewpoint corrected color image and a multi-viewpoint parallax related image, and performs multi-viewpoint encoding processing to add viewpoint generation information and the like.
- the details of this multi-view encoding process will be described with reference to FIG. 6 described later.
- step S19 the multiview image encoding unit 15 transmits the encoded bit stream generated as a result of step S18, and ends the process.
- FIG. 6 is a flowchart for explaining the details of the multi-view encoding process of step S18 of FIG. This multiview coding process is performed, for example, in slice units. Further, in the multi-viewpoint encoding process of FIG. 6, it is assumed that the image to be encoded is a color image A, a color image B, a parallax related image A, and a parallax related image B.
- step S31 of FIG. 6 the multi-viewpoint image encoding unit 15 generates an SPS of a target slice that is a slice to be processed, and assigns a unique ID to the SPS.
- step S32 the multi-viewpoint image encoding unit 15 generates a Subset SPS of the target slice, and assigns a unique ID to the Subset SPS.
- step S33 the multi-viewpoint image encoding unit 15 generates PPS of the target slice including the ID assigned in steps S31 and S32, and assigns a unique ID to the PPS.
- step S34 the multi-viewpoint image encoding unit 15 generates SEI including viewpoint generation information of the target slice.
- step S35 the multi-viewpoint image encoding unit 15 encodes the target slice of the color image A as a base view, and adds a header portion including the ID assigned in step S33.
- step S36 the multiview image encoding unit 15 encodes the target slice of the parallax related image A as a non-base view, and adds a header portion including the ID assigned in step S33.
- step S37 the multiview image encoding unit 15 encodes the target slice of the color image B as a non-base view, and adds a header portion including the ID assigned in step S33.
- step S38 the multiview image encoding unit 15 encodes the target slice of the parallax related image B as a non-base view, and adds a header portion including the ID assigned in step S33.
- the multi-viewpoint image encoding unit 15 generates the SPS, Subset SPS, PPS, SEI, target slice of color image A, target slice of parallax related image A, target slice of color image B, and parallax related image B.
- the target slices are arranged in order to generate a coded bit stream. Then, the process returns to step S18 in FIG. 5 and proceeds to step S19.
- SPS is generated in slice units, but if the SPS of the current target slice is the same as the SPS of the previous target slice, SPS is not generated.
- SPS is not generated.
- Subset SPS, PPS, and SEI is generated in slice units, but if the SPS of the current target slice is the same as the SPS of the previous target slice.
- the encoding device 10 generates viewpoint generation information according to a predetermined method of generating color images of other viewpoints using the multiview corrected color image and the multiview parallax related image, and It transmits with the color image and parallax related image of the viewpoint of. Therefore, in the decoding device described later, a color image of a viewpoint different from that of the viewpoint can be generated using a color image of a predetermined viewpoint, a parallax related image, and information for viewpoint generation.
- FIG. 7 is a block diagram showing a configuration example of a first embodiment of a decoding device as an image processing device to which the present technology is applied, which decodes a coded bit stream transmitted from the coding device 10 of FIG. .
- the decoding device 30 in FIG. 7 includes a multi-viewpoint image decoding unit 31, a viewpoint combination unit 32, and a multi-viewpoint image display unit 33.
- the multi-view image decoding unit 31 of the decoding device 30 functions as a receiving unit, and receives the coded bit stream transmitted from the coding device 10 of FIG.
- the multi-viewpoint image decoding unit 31 extracts viewpoint generation information from the SEI of the received encoded bit stream, and supplies the information to the viewpoint combining unit 32.
- the multi-view image decoding unit 31 functions as a decoding unit, decodes the coded bit stream by a method corresponding to the coding method of the multi-view image coding unit 15 in FIG. A multi-viewpoint parallax related image is generated.
- the multi-viewpoint image decoding unit 31 supplies the generated multi-viewpoint corrected color image and the multi-viewpoint parallax related image to the viewpoint combining unit 32.
- the viewpoint combining unit 32 uses the viewpoint generation information from the multiview image decoding unit 31 to set the number of viewpoints corresponding to the multiview image display unit 33 to the multiview parallax related image from the multiview image decoding unit 31. Warping processing to the display viewpoint of (described in detail later) is performed.
- the viewpoint combining unit 32 selects one of the multi-view corrected color image and the multi-view disparity related image supplied from the multi-view image decoding unit 31 based on the disparity related image specifying information included in the viewpoint generation information.
- the multi-viewpoint parallax related image of the viewpoint for the number of viewpoints of the parallax related image is identified.
- the viewpoint combining unit 32 acquires the internal parameter from the viewpoint generation information.
- the viewpoint synthesis unit 32 acquires the external parameter from the viewpoint generation information.
- the viewpoint combining unit 32 warps the multi-viewpoint parallax related image to the display viewpoint based on the acquired internal parameters and external parameters, and the image type information and the range information included in the viewpoint generation information. I do.
- the warping process is a process of geometrically transforming an image of one viewpoint to an image of another viewpoint.
- the display viewpoint includes a viewpoint other than the viewpoint corresponding to the multi-view color image.
- the viewpoint combining unit 32 uses the viewpoint generation information and the parallax related image of the display viewpoint obtained as a result of the warping process to the multi-view corrected color image supplied from the multi-view image decoding unit 31. Perform warping processing to the display viewpoint. Specifically, the viewpoint combining unit 32 selects one of the multi-view corrected color image and the multi-view parallax related image supplied from the multi-view image decoding unit 31 based on the color image identification information included in the viewpoint generation information. , Identify multi-view color images of the same number of viewpoints as color images. Then, using the parallax related image of the display viewpoint obtained as a result of the warping process, the viewpoint combining unit 32 performs a warping process to the display viewpoint on the multi-viewpoint corrected color image.
- the viewpoint combining unit 32 supplies the color image of the display viewpoint obtained as a result to the multi-viewpoint image display unit 33 as a multi-viewpoint combined color image.
- the multi-viewpoint image display unit 33 displays the multi-viewpoint composite color image supplied from the viewpoint combining unit 32 such that the viewable angle differs for each viewpoint.
- the viewer can view 3D images from a plurality of viewpoints without wearing glasses by viewing the images of any two viewpoints with the left and right eyes.
- FIG. 8 is a diagram for explaining the warping process of the depth image.
- the depth value (value y) of the depth image of the viewpoint at the position t (t x , t y , t z ) is the position Z (depth in the depth direction on world coordinates) according to the following equation (2)
- a position M (X, Y, Z) on the world coordinates corresponding to the position m (x, y, z) on the screen of the pixel corresponding to Z) is determined, and the position M (X, Y)
- the position m ′ (x ′, y ′, z ′) on the screen of the depth image of the display viewpoint at the position t ′ (t ′ x , t ′ y , t ′ z ) corresponding to L, Z) is obtained.
- R is a rotation matrix for converting coordinates on the multi-viewpoint color image capturing unit 11 into world coordinates, and is a predetermined fixed matrix.
- R ′ is a rotation matrix for converting coordinates on a virtual imaging unit for capturing a color image corresponding to a depth image after warping processing into world coordinates.
- A is a matrix for converting the coordinates on the multi-viewpoint color image capturing unit 11 into the coordinates on the screen, and is expressed by the following equation (3).
- focal_length_x and focal_length_y respectively indicate focal lengths in the horizontal direction and the vertical direction (y direction) of the multi-viewpoint color image capturing unit 11.
- principal_point_x and principal_point_y respectively indicate horizontal and vertical positions of principal points of the multi-view color image capturing unit 11.
- radial_distortion represents a distortion factor in the radial direction.
- the focal length in the vertical direction and the position in the vertical direction of the principal point are not used in Equation (3). Further, the correction is performed in the multi-viewpoint color image correction unit 12 so that the distortion coefficient in the radial direction becomes zero.
- a ′ is a matrix for converting coordinates on a virtual imaging unit for capturing a color image corresponding to a warped depth image, which is expressed similarly to A, to coordinates on the screen, It is decided beforehand.
- s is a scaling factor, which is a predetermined fixed value.
- t y and t z are predetermined fixed values.
- information other than the depth image which is necessary for correlating the position m (x, y, z) and the position m '(x', y ', z'), is the horizontal focal length, the horizontal direction of the principal point , The horizontal position of the multi-view color image capturing unit 11, and the minimum depth value and the maximum depth value required to convert the depth value into the position Z. And, these pieces of information are included in the viewpoint generation information.
- the depth image to be processed based on the position m '(x', y ', z') corresponding to the position m (x, y, z) of each pixel
- the pixel of the depth image after the warping process corresponding to each pixel of is determined.
- the depth value of each pixel of the depth image to be processed is taken as the depth value of the pixel of the depth image after the warping process corresponding to the pixel.
- the warping process on the parallax image is performed in the same manner as the warping process on the depth image except that the depth value (value y) is replaced with the parallax value (value I).
- FIG. 9 is a flowchart illustrating the decoding process of the decoding device 30 of FIG. This decoding process is started, for example, when a coded bit stream is transmitted from the coding apparatus 10 of FIG.
- step S51 of FIG. 9 the multi-viewpoint image decoding unit 31 of the decoding device 30 receives the coded bit stream transmitted from the coding device 10 of FIG.
- step S52 the multi-viewpoint image decoding unit 31 decodes the received encoded bit stream, and performs multi-viewpoint decoding processing to extract viewpoint generation information. The details of this multi-viewpoint decoding process will be described with reference to FIG. 10 described later.
- the view synthesis unit 32 functions as a generation unit, and performs multiview synthesis using the viewpoint generation information supplied from the multiview image decoding unit 31, the multiview corrected color image, and the multiview parallax related image. Generate a color image.
- step S54 the multi-viewpoint image display unit 33 displays the multi-viewpoint composite color image supplied from the viewpoint combining unit 32 so that the viewable angle differs for each viewpoint, and the process ends.
- FIG. 10 is a flowchart for explaining the details of the multiview decoding process of step S52 of FIG. This multi-viewpoint decoding process is performed, for example, in slice units. Further, in the multiview decoding process of FIG. 10, it is assumed that the image to be decoded is a color image A, a color image B, a parallax related image A, and a parallax related image B.
- step S71 of FIG. 10 the multi-viewpoint image decoding unit 31 extracts an SPS from the received coded bit stream.
- step S72 the multi-viewpoint image decoding unit 31 extracts Subset SPS from the encoded bit stream.
- step S73 the multiview image decoding unit 31 extracts the PPS from the coded bit stream.
- step S74 the multi-viewpoint image decoding unit 31 extracts the SEI from the encoded bit stream, and supplies the viewpoint generation unit 32 with the viewpoint generation information included in the SEI.
- step S75 the multi-viewpoint image decoding unit 31 activates the PPS to which the ID is added, based on the PPS ID included in the header of the target slice of the color image A.
- step S76 the multi-viewpoint image decoding unit 31 activates the SPS to which the ID is added based on the ID of the SPS included in the PPS activated in step S75.
- step S77 the multi-viewpoint image decoding unit 31 decodes the target slice of the color image A as a base view with reference to the PPS and SPS activated, and supplies the decoded image to the viewpoint combining unit 32.
- step S78 the multi-viewpoint image decoding unit 31 activates the Subset SPS to which the ID is assigned, based on the ID of the Subset SPS included in the PPS being activated.
- step S 79 the multi-viewpoint image decoding unit 31 decodes the target slice of the parallax related image A as a non-base view with reference to the PPS and Subset SPS that are activated, and supplies the non-base view to the viewpoint combination unit 32.
- step S80 the multi-viewpoint image decoding unit 31 decodes the target slice of the color image B as a non-base view with reference to the activated PPS and Subset SPS, and supplies the non-base view to the viewpoint combination unit 32.
- step S81 the multi-viewpoint image decoding unit 31 decodes the target slice of the parallax related image B as a non-base view with reference to the PPS and Subset SPS that are activated, and supplies the non-base view to the viewpoint combination unit 32. Then, the process returns to step S52 in FIG. 9 and proceeds to step S53.
- SPS, Subset SPS, PPS, and SEI are generated for all slices for convenience of explanation, and are always extracted for each slice. If there is a slice for which SPS, Subset SPS, PPS, SEI is not generated, the process of extracting the SPS, Subset SPS, PPS, SEI is skipped.
- the decoding device 30 receives viewpoint generation information from the encoding device 10 together with the color image and the parallax related image of the predetermined viewpoint. Therefore, the decoding device 30 can generate a color image of a viewpoint different from the viewpoint using the color image of the predetermined viewpoint, the parallax related image, and the viewpoint generation information.
- a color image of a predetermined viewpoint and a parallax related image are used to generate a color image of a viewpoint different from that viewpoint. It does not contain the necessary information. Specifically, information for identifying a color image and a parallax related image, information for identifying a parallax image and a depth image, and range information are not included.
- the Multiview acquisition information SEI shown in FIG. 1 includes parameters inside the camera and outside the camera, there is much unnecessary information other than the viewpoint generation information. Furthermore, in Multiview acquisition information SEI, parameters in the camera are described for all viewpoints regardless of the type of parameter or only for specific viewpoints. Therefore, when parameters in the camera are described for all viewpoints, there is a lot of redundant information, and when only specific viewpoints are described, the information is insufficient. In addition, in Multiview acquisition information SEI, since parameters outside the camera are described for all viewpoints, there is much redundant information.
- the multi-viewpoint parallax related image is generated from the multi-viewpoint corrected color image in the encoding device 10
- the image may be generated by a sensor that detects a parallax value or a depth value when capturing the multi-view color image. The same applies to the encoding device 50 described later.
- the viewpoint generation information may include only one of the color image specification information and the parallax related image specification information.
- the decoding device 30 specifies the unidentified one of the multi-view color image and the multi-view parallax related image as the unidentified image. For example, when only the color image identification information is included in the viewpoint generation information, the decoding device 30 identifies the multi-view color image based on the color image identification information, and identifies the other images as the multi-view parallax related image Do.
- the viewpoint generation information is included in the SEI and transmitted.
- the viewpoint generation information is a video coding layer (VCL) or an SPS (Sequence) of a network abstraction layer (NAL). It can also be transmitted by being included in Parameter Set), PPS (Picture Parameter Set) or the like.
- FIG. 11 is a block diagram showing a configuration example of a second embodiment of an encoding device as an image processing device to which the present technology is applied.
- the encoding device 50 in FIG. 11 includes a multiview color image capturing unit 51, a multiview color image correction unit 52, a multiview parallax related image generation unit 53, a viewpoint generation information generation unit 54, and a multiview image coding unit 55. It consists of The encoding device 50 transmits part of the viewpoint generation information as information on encoding (encoding parameter).
- the multi-view color image capturing unit 51 of the encoding device 10 captures a multi-view color image and supplies the multi-view color image correction unit 52 as a multi-view color image.
- the multi-view color image capturing unit 51 generates external parameters, disparity-related maximum values (depth maximum values), and disparity-related minimum values (depth minimum values) (details will be described later).
- the multi-view color image capturing unit 51 supplies the external parameters, the disparity-related maximum value, and the disparity-related minimum value to the viewpoint generation information generating unit 54, and the disparity-related maximum value and the disparity-related minimum value are multi-view disparity related images
- the data is supplied to the generation unit 53.
- the parallax-related maximum value is a depth maximum value when the parallax-related image generated by the multi-viewpoint parallax-related image generation unit 53 is a depth image, and is a parallax maximum value when the parallax-related image is a parallax image.
- the parallax-related minimum value is a depth minimum value when the parallax-related image generated by the multi-view parallax-related image generation unit 53 is a depth image, and is a parallax minimum value when the parallax-related image is a parallax image.
- the multi-viewpoint color image correction unit 52 performs color correction, brightness correction, distortion correction, and the like on the multi-viewpoint color image supplied from the multi-viewpoint color image pickup unit 51. As a result, the focal length in the horizontal direction (X direction) of the multi-view color image capturing unit 51 in the corrected multi-view color image becomes common to all the viewpoints.
- the multi-view color image correction unit 52 supplies the multi-view color image after correction as a multi-view correction color image to the multi-view parallax related image generation unit 53 and the multi-view image coding unit 55.
- the multiview parallax related image generation unit 53 generates the multiview corrected color image supplied from the multiview color image correction unit 52 based on the parallax associated maximum value and the parallax associated minimum value supplied from the multiview color image capturing unit 51. , To generate parallax-related images of multiple viewpoints. Specifically, the multi-view parallax related image generation unit 53 generates parallax related values (inverse 1 / Z of the depth Z or parallax d) before normalization of each pixel from the multi-view corrected color image for each of the multi-view points. And the disparity related value before normalization is normalized based on the disparity related maximum value and the disparity related minimum value.
- the parallax-related image generation unit 53 sets parallax-related values (value y and value I) of each pixel normalized as the pixel value of each pixel of the parallax-related image for each viewpoint of the multi-viewpoint. Generate an image.
- the multi-viewpoint parallax related image generation unit 53 supplies the generated multi-viewpoint parallax related image to the multi-viewpoint image encoding unit 55 as a multi-viewpoint parallax related image. Further, the multi-viewpoint parallax related image generation unit 53 generates a parallax accuracy parameter (depth accuracy parameter) representing the accuracy of the pixel value of the multi-viewpoint parallax related image, and supplies the parallax accuracy parameter to the viewpoint generation information generation unit 54.
- a parallax accuracy parameter depth accuracy parameter
- the viewpoint generation information generation unit 54 functions as a generation unit, and generates viewpoint generation information according to a predetermined method of generating a color image of another viewpoint using the multi-viewpoint corrected color image and the multi-viewpoint parallax related image. Generate Specifically, the viewpoint generation information generation unit 54 obtains the inter-camera distance based on the external parameter supplied from the multi-view color image capturing unit 51.
- the inter-camera distance corresponds to the horizontal position of the multi-view color image capturing unit 51 when capturing a color image of the multi-view parallax related image and the color image and the parallax related image when capturing a color image of the viewpoint. It is the distance of the position of the horizontal direction of the multi-view color image capturing unit 51 when capturing a color image having parallax.
- the viewpoint generation information generation unit 54 generates viewpoint generation parameters for the parallax-related maximum value and the parallax-related minimum value from the multi-view color image capturing unit 51, the inter-camera distance, and the parallax accuracy parameters from the multi-view parallax related image generation unit 53. It is information.
- the viewpoint generation information generation unit 54 supplies the generated viewpoint generation information to the multi-viewpoint image coding unit 55.
- the multi-viewpoint image encoding unit 55 functions as an encoding unit, and performs the multi-viewpoint correction color image from the multi-viewpoint color image correction unit 52 and the multi-viewpoint parallax related image from the multiview parallax related image generation unit 53 in HEVC (High). Coding is performed according to the Efficiency Video Coding method. As for the HEVC system, as of August 2011, as Draft, Thomas Wiegand, Woo-jin Han, Benjamin Bross, Jens-Rainer Ohm, Gary J. Sullivian, "WD3: Working Draft 3 of High-Efficiency Video Coding", JCTVC -E603_d5 (version 5), May 20, 2011 has been issued.
- the multi-viewpoint image encoding unit 55 differentially encodes the parallax-related maximum value, the parallax-related minimum value, and the inter-camera distance among the viewpoint generation information supplied from the viewpoint generation information generation unit 54, and performs multiview. It is included in the information on the encoding of disparity related images. Then, the multi-view image encoding unit 55 performs encoding including the encoded multi-view corrected color image and multi-view disparity related image, differentially encoded disparity related maximum value, disparity related minimum value, and inter-camera distance. And a bit stream including parallax accuracy parameters and the like from the viewpoint generation information generation unit 54 as a coded bit stream.
- the multi-viewpoint image encoding unit 55 differentially encodes and transmits the disparity-related maximum value, the disparity-related minimum value, and the inter-camera distance, so the code amount of the viewpoint generation information can be reduced. .
- differential encoding is effective in reducing the amount of code because disparity-related maximum value, disparity-related minimum value, and inter-camera distance are likely not to change significantly between pictures. It is.
- FIG. 12 is a diagram for explaining the parallax related maximum value and the parallax related minimum value of the viewpoint generation information.
- the horizontal axis is the parallax related value before normalization
- the vertical axis is the pixel value (depth information) of the parallax related image.
- the multi-view parallax related image generation unit 53 generates the parallax related value (reciprocal number 1 / Z of the depth Z or the parallax d) before normalization of each pixel, the parallax related minimum value Dmin and the parallax related maximum For example, the value Dmax is normalized to a value of 0 to 255. Then, the multi-viewpoint parallax related image generation unit 53 generates a parallax related image using the parallax related value (depth information) of each pixel after normalization which is a value of 0 to 255 as a pixel value.
- the pixel value I of each pixel of the parallax related image is described above using the parallax related value d (parallax d) before normalization of the pixel, the parallax related minimum value D min , and the parallax related maximum value D max .
- d parallax related value before normalization of the pixel
- D min parallax related minimum value
- D max parallax related maximum value
- the disparity before normalization is performed using the disparity related minimum value D min and the disparity related maximum value D max from the pixel value I of each pixel of the disparity related image according to the following equation (5) It is necessary to restore the related value d.
- the pixel value y of each pixel of the parallax related image is calculated using the parallax related value Z (parallax Z) before normalization of that pixel, the parallax related minimum value Z near , and the parallax related maximum value Z far as described above. As shown, it is expressed by the equation (c). Therefore, in the decoding device described later, it is necessary to restore the parallax related value Z from the pixel value y of each pixel of the parallax related image using the parallax related minimum value Z near and the parallax related maximum value Z far . Therefore, the disparity related minimum value and the disparity related maximum value are transmitted to the decoding device.
- FIG. 13 is a diagram for explaining the parallax accuracy parameter of the viewpoint generation information.
- disparity accuracy As shown in the upper part of FIG. 13, when the disparity-related value before normalization (the reciprocal 1 / Z of depth Z or the disparity d) per disparity-related value (depth information) 1 after normalization is 0.5, disparity accuracy The parameter represents the accuracy 0.5 of the disparity related value (depth information) after normalization. In addition, as shown in the lower part of FIG. 13, when the disparity related value before normalization per disparity related value 1 after normalization is 1, the disparity accuracy parameter represents the accuracy 1.0 of the disparity related value. .
- the disparity related value before normalization of the first view point of view # 1 is 1.0
- the disparity related value of the second view point of view # 2 before normalization is 0.5
- the disparity related value after normalization of the viewpoint # 1 is 1.0 regardless of whether the accuracy of the disparity related value is 0.5 or 1.0.
- the parallax related value of the viewpoint # 2 is 0.5 when the accuracy of the parallax related value is 0.5, and is 0 when the accuracy of the parallax related value is 1.0.
- FIG. 14 is a diagram for explaining the inter-camera distance of the viewpoint generation information.
- the inter-camera distance of the parallax related image of viewpoint # 1 based on viewpoint # 2 is the distance between the position represented by the external parameter of viewpoint # 1 and the position represented by the external parameter of viewpoint # 2 It is.
- FIG. 15 is a block diagram showing a configuration example of the multi-viewpoint image coding unit 55 of FIG.
- the multi-viewpoint image coding unit 55 of FIG. 15 is configured of a slice coding unit 61, a slice header coding unit 62, a PPS coding unit 63, and an SPS coding unit 64.
- the slice encoding unit 61 of the multiview image encoding unit 55 performs the multiview corrected color image supplied from the multiview color image correction unit 52 and the multiview parallax related image supplied from the multiview parallax related image generation unit 53. Are encoded in a scheme according to the HEVC scheme in slice units.
- the slice encoding unit 61 supplies the slice header encoding unit 62 with the encoded data in slice units obtained as a result of encoding.
- the slice header encoding unit 62 sets the parallax-related maximum value, the parallax-related minimum value, and the inter-camera distance among the viewpoint generation information supplied from the viewpoint generation information generation unit 54 to the parallax of the current processing target slice. As the related maximum value, the parallax related minimum value, and the inter-camera distance, they are held.
- the slice header encoding unit 62 is configured so that the disparity-related maximum value, disparity-related minimum value, and inter-camera distance of the current processing target slice are disparity-related one slice earlier in the coding order than that slice. It is determined in the unit to which the same PPS is added (hereinafter referred to as the same PPS unit) whether or not the maximum value, the parallax related minimum value, and the inter-camera distance match.
- the parallax related maximum value, the parallax related minimum value, and the inter-camera distance of all slices constituting the same PPS unit are the parallax related maximum value, the parallax related minimum value, and the camera of the immediately preceding slice in coding order, and the camera
- the slice header encoding unit 62 determines, as a slice header of encoded data of each slice forming the same PPS unit, the disparity related maximum value, disparity related minimum value of the slice, and Information on encoding other than the inter-camera distance is added and supplied to the PPS encoding unit 63.
- the slice header encoding unit 62 supplies the PPS encoding unit 63 with a transmission flag indicating no transmission of the disparity related maximum value, the disparity related minimum value, and the differential encoding result of the inter-camera distance.
- the disparity-related maximum value, disparity-related minimum value, and inter-camera distance of at least one slice constituting the same PPS unit are the disparity-related maximum value, disparity-related minimum value and disparity-related minimum value of the immediately preceding slice in coding order, and If it is determined that the inter-camera distance does not match, the slice header encoding unit 62 sets the disparity related maximum value of the slice, the disparity related minimum value, and the camera as a slice header in the encoded data of the intra type slice. Information on coding including inter-distance is added and supplied to the PPS coding unit 63.
- the slice header encoding unit 62 differentially encodes the disparity related maximum value, disparity related minimum value, and inter-camera distance of an inter-type slice. Specifically, the slice header encoding unit 62 calculates the disparity related maximum of the slice immediately preceding in slice in coding order from the disparity related maximum value, disparity related minimum value, and inter-camera distance of the inter type slice. The value, the disparity-related minimum value, and the inter-camera distance are respectively subtracted to obtain a differential encoding result.
- the slice header encoding unit 62 adds information related to encoding including the disparity related maximum value, the disparity related minimum value, and the differential encoding result of the inter-camera distance as a slice header to the encoded data of the inter type slice , And supplies it to the PPS encoding unit 63.
- the slice header encoding unit 62 supplies the PPS encoding unit 63 with a transmission flag indicating the presence of transmission of the parallax associated maximum value, the parallax associated minimum value, and the differential encoding result of the inter-camera distance.
- the PPS encoding unit 63 includes a PPS including the transmission flag supplied from the slice header encoding unit 62 and the parallax accuracy parameter in the viewpoint generation information supplied from the viewpoint generation information generation unit 54 of FIG. Generate The PPS encoding unit 63 adds the PPS to the slice unit encoded data to which the slice header supplied from the slice header encoding unit 62 is added in the same PPS unit, and supplies the PPS to the SPS encoding unit 64.
- the SPS encoding unit 64 generates an SPS. Then, the SPS encoding unit 64 adds the SPS to the encoded data to which the PPS supplied from the PPS encoding unit 63 is added, in sequence units. The SPS encoding unit 64 transmits the resultant bit stream as a coded bit stream.
- FIG. 16 is a diagram showing an exemplary configuration of a coded bit stream.
- the encoded bit stream includes the encoded data of the slice of the multiview color image Will also be placed.
- PPS # 0 includes a transmission flag "1" indicating the presence of transmission.
- the parallax accuracy of the slices constituting the same PPS unit of PPS # 0 is 0.5
- PPS # 0 includes “1” representing the parallax accuracy of 0.5 as the parallax accuracy parameter.
- the disparity related minimum value of the intra type slice constituting the same PPS unit of PPS # 0 is 10
- the disparity related maximum value is 50
- the inter-camera distance is 100. Therefore, the slice header of the slice includes the disparity related minimum value “10”, the disparity related maximum value “50”, and the inter-camera distance “100”.
- the parallax related minimum value of the first inter type slice constituting the same PPS unit of PPS # 0 is 9
- the parallax related maximum value is 48
- the inter-camera distance is 105. is there. Therefore, in the slice header of the slice, the difference “ ⁇ 1” obtained by subtracting the disparity related minimum value “10” of the slice of the previous intra type in coding order from the disparity related minimum value “9” of the slice Are included as a difference encoding result of the disparity related minimum value.
- the difference “-2” of the parallax related maximum value is included as the difference encoding result of the parallax related maximum value
- the difference “5” of the inter-camera distance is included as the difference coding result of the inter-camera distance.
- the disparity related minimum value of the second inter type slice constituting the same PPS unit of PPS # 0 is 7
- the disparity related maximum value is 47
- the inter-camera distance is 110. is there. Therefore, in the slice header of the slice, a difference “9” of the disparity-related minimum value “9” of the first inter-type slice in the coding order is subtracted from the disparity-related minimum value “7” of the slice. “ ⁇ 2” is included as the difference encoding result of the disparity related minimum value.
- the difference “ ⁇ 1” of the disparity related maximum value is included as the difference encoding result of the disparity related maximum value
- the difference “5” of the inter-camera distance is included as the difference encoding result of the inter-camera distance.
- disparity related maximum values, disparity related minimum values, and inter-camera values of one intra-type slice and two inter-type slices constituting the same PPS unit of the first PPS PPS # The distances correspond to the disparity-related maximum value, disparity-related minimum value, and inter-camera distance of the immediately preceding slice in coding order, respectively. That is, the disparity-related minimum value, disparity-related maximum value, and inter-camera distance of one intra-type slice and two inter-type slices constituting the same PPS unit of PPS # 1 are respectively the same PPS unit of PPS # 0.
- PPS # 1 includes a transmission flag "0" indicating that there is no transmission. Further, in the example of FIG. 16, the parallax accuracy of the slices constituting the same PPS unit of PPS # 1 is 0.5, and PPS # 1 includes “1” representing the parallax accuracy of 0.5 as the parallax accuracy parameter.
- FIG. 17 is a diagram showing an example of PPS syntax of FIG.
- the PPS includes a parallax accuracy parameter (disparity_precision) and a transmission flag (dsiparity_pic_same_flag).
- the parallax accuracy parameter is, for example, “0” when representing parallax accuracy (the accuracy of depth information) 1 and “2” when representing a parallax accuracy of 0.25. Further, as described above, the parallax accuracy parameter is “1” when representing the parallax accuracy of 0.5. Further, as described above, the transmission flag is “1” when indicating the presence of transmission, and is “0” when indicating the absence of transmission.
- FIG. 18 and FIG. 19 are diagrams showing examples of slice header syntax.
- the transmission flag is 1 and the slice type is intra type
- the disparity related minimum value (minimum_disparity)
- disparity related maximum value (maximum_disparity)
- the inter-camera distance contains translation_x).
- the slice header contains the difference encoding result of the disparity related minimum value (delta_minimum_disparity), the difference encoding result of the disparity related maximum value (delta_maximum_disparity), And the inter-camera distance differential encoding result (delta_translation_x).
- FIG. 20 is a flowchart illustrating the encoding process of the encoding device 50 of FIG.
- step S111 in FIG. 20 the multi-view color image capturing unit 51 of the encoding device 50 captures a multi-view color image and supplies the multi-view color image correction unit 52 as a multi-view color image.
- step S112 the multi-view color image capturing unit 51 generates disparity-related maximum values, disparity-related minimum values, and external parameters.
- the multi-view color image capturing unit 51 supplies the disparity-related maximum value, the disparity-related minimum value, and the external parameter to the viewpoint generation information generating unit 54, and the disparity-related maximum value and the disparity-related minimum value are multi-view disparity related images
- the data is supplied to the generation unit 53.
- step S113 the multiview color image correction unit 52 performs color correction, luminance correction, distortion correction, and the like on the multiview color image supplied from the multiview color image capturing unit 51.
- the focal length in the horizontal direction (X direction) of the multi-view color image capturing unit 51 in the corrected multi-view color image becomes common to all the viewpoints.
- the multi-view color image correction unit 52 supplies the multi-view color image after correction as a multi-view correction color image to the multi-view parallax related image generation unit 53 and the multi-view image coding unit 55.
- step S114 the multi-viewpoint parallax related image generation unit 53 supplies the multi-view color image correction unit 52 based on the parallax-related maximum value and the parallax-related minimum value supplied from the multi-view color image capturing unit 51.
- a parallax-related image of multiple viewpoints is generated from the viewpoint corrected color image.
- the multi-viewpoint parallax related image generation unit 53 supplies the generated multi-viewpoint parallax related image to the multi-viewpoint image encoding unit 55 as a multi-viewpoint parallax related image.
- step S115 the multi-viewpoint parallax related image generation unit 53 generates a parallax accuracy parameter, and supplies the parallax accuracy parameter to the viewpoint generation information generation unit 54.
- step S116 the viewpoint generation information generation unit 54 obtains the inter-camera distance based on the external parameter supplied from the multi-view color image capturing unit 51.
- step S117 the viewpoint generation information generation unit 54 generates the parallax related maximum value and the parallax related minimum value from the multi-viewpoint color image capturing unit 51, the inter-camera distance, and the parallax accuracy parameter from the multi-viewpoint parallax related image generating unit 53. Is generated as viewpoint generation information.
- the viewpoint generation information generation unit 54 supplies the generated viewpoint generation information to the multi-viewpoint image coding unit 55.
- the multi-viewpoint image encoding unit 55 is a scheme based on the HEVC scheme for the multi-viewpoint corrected color image from the multi-viewpoint color image correction unit 52 and the multi-viewpoint parallax related image from the multiview parallax related image generation unit 53. Perform multi-view coding processing for coding in Details of this multi-view encoding process will be described with reference to FIG. 21 described later.
- step S119 the multiview image encoding unit 55 transmits the encoded bit stream obtained as a result of the multiview encoding process, and ends the process.
- FIG. 21 is a flow chart for explaining the multi-view coding process of step S118 of FIG.
- the slice encoding unit 61 (FIG. 15) of the multi-viewpoint image encoding unit 55 compares the multi-viewpoint corrected color image from the multi-viewpoint color image correction unit 52 with the multi-viewpoint parallax related image generation unit 53.
- the multi-view parallax related image from is encoded in a scheme according to the HEVC scheme on a slice basis.
- the slice encoding unit 61 supplies the slice header encoding unit 62 with the encoded data in slice units obtained as a result of encoding.
- step S132 the slice header encoding unit 62 processes the inter-camera distance, the parallax-related maximum value, and the parallax-related minimum value among the viewpoint generation information supplied from the viewpoint generation information generation unit 54 as the current processing target.
- the inter-camera distance of the slice, the parallax-related maximum value, and the parallax-related minimum value are held.
- step S133 the slice header encoding unit 62 determines that the inter-camera distances, disparity-related maximum values, and disparity-related minimum values of all slices constituting the same PPS unit are each one in coding order before that slice. It is determined whether the inter-camera distance of the slice, the disparity-related maximum value, and the disparity-related minimum value match.
- step S134 the slice header encoding unit 62 determines the inter-camera distance, disparity-related maximum value, and disparity-related minimum A transmission flag representing the absence of transmission of the differential encoding result of values is generated and supplied to the PPS encoding unit 63.
- step S135 the slice header encoding unit 62 sets, as a slice header, the encoded data of each slice constituting the same PPS unit to be processed in step S133, the inter-camera distance of that slice, the disparity related maximum value, Information on coding other than disparity related minimum value is added. Then, the slice header encoding unit 62 supplies the encoded data of each slice constituting the same PPS unit obtained as a result to the PPS encoding unit 63, and the process proceeds to step S140.
- step S136 when it is determined in step S133 that the inter-camera distance, the parallax associated maximum value, and the parallax associated minimum value do not match, in step S136, the slice header encoding unit 62 determines the inter camera distance, the parallax associated maximum value, and A transmission flag indicating the presence of transmission of the difference encoding result of the disparity related minimum value is supplied to the PPS encoding unit 63.
- the processing of steps S137 to S139 described later is performed for each slice constituting the same PPS unit which is the processing target of step S133.
- step S137 the slice header encoding unit 62 determines whether the type of slice forming the same PPS unit to be processed in step S133 is an intra type.
- step S138 the slice header encoding unit 62 sets the slice inter-camera distance and disparity relation as a slice header to the encoded data of the slice. Add information about the coding including maximum value and disparity related minimum value. Then, the slice header encoding unit 62 supplies the encoded data in slice units obtained as a result to the PPS encoding unit 63, and the process proceeds to step S140.
- step S137 if it is determined in step S137 that the slice type is not the intra type, that is, if the slice type is the inter type, the process proceeds to step S139.
- step S139 the slice header encoding unit 62 differentially encodes the inter-camera distance of the slice, the disparity-related maximum value, and the disparity-related minimum value, and encodes the difference encoding result in the encoded data of the slice. The information about is added as a slice header. Then, the slice header encoding unit 62 supplies the encoded data in slice units obtained as a result to the PPS encoding unit 63, and the process proceeds to step S140.
- step S140 the PPS encoding unit 63 uses the transmission flag supplied from the slice header encoding unit 62 and the parallax accuracy parameter of the viewpoint generation information supplied from the viewpoint generation information generation unit 54 of FIG. Generate the included PPS.
- step S 141 the PPS encoding unit 63 adds the PPS to the slice unit encoded data to which the slice header supplied from the slice header encoding unit 62 is added in the same PPS unit, and transmits the PPS to the SPS encoding unit 64. Supply.
- step S142 the SPS encoding unit 64 generates an SPS.
- step S143 the SPS encoding unit 64 adds the SPS to the encoded data to which the PPS supplied from the PPS encoding unit 63 is added in sequence units, and generates an encoded bit stream.
- the encoding device 50 arranges the inter-camera distance, the disparity-related maximum value, and the disparity-related minimum value in the slice header as information regarding encoding. This makes it possible, for example, to use the inter-camera distance, the disparity-related maximum value, and the disparity-related minimum value for encoding.
- FIG. 22 is a block diagram showing a configuration example of a second embodiment of a decoding device as an image processing device to which the present technology is applied, which decodes a coded bit stream transmitted from the coding device 50 of FIG. .
- the configuration of the decoding device 80 in FIG. 22 is different from the configuration in FIG. 7 in that a multi-view image decoding unit 81 and a viewpoint combining unit 82 are provided instead of the multi-view image decoding unit 31 and the viewpoint combining unit 32.
- the decoding device 80 displays a multi-view composite color image based on the viewpoint generation information transmitted from the encoding device 50.
- the multi-viewpoint image decoding unit 81 of the decoding device 80 functions as a receiving unit, and receives the coded bit stream transmitted from the coding device 50 of FIG.
- the multi-viewpoint image decoding unit 81 extracts the parallax accuracy parameter and the transmission flag from the PPS included in the received encoded bit stream. Further, the multi-viewpoint image decoding unit 81 extracts the inter-camera distance, the disparity related maximum value, and the disparity related minimum value from the slice header of the coded bit stream according to the transmission flag.
- the multi-viewpoint image decoding unit 81 generates viewpoint generation information including the parallax accuracy parameter, the inter-camera distance, the parallax-related maximum value, and the parallax-related minimum value, and supplies the information to the viewpoint synthesis unit 82.
- the multi-view image decoding unit 81 functions as a decoding unit, and uses a scheme corresponding to the encoding scheme of the multi-view image encoding unit 15 in FIG. 11 on slice unit encoded data included in the encoded bit stream. Decode to generate a multi-view corrected color image and a multi-view disparity related image.
- the multi-viewpoint image decoding unit 81 supplies the generated multi-viewpoint corrected color image and the multi-viewpoint parallax related image to the viewpoint combining unit 82.
- the viewpoint combining unit 82 performs warping processing on the multi-viewpoint parallax related image from the multi-viewpoint image decoding unit 81 to the display viewpoint using the viewpoint generation information from the multi-view image decoding unit 81. Specifically, the viewpoint combining unit 82 determines whether the multi-viewpoint parallax is related to the parallax accuracy parameter based on the inter-camera distance, the parallax-related maximum value, the parallax-related minimum value, and the like included in the viewpoint generation information. Warping processing to the display viewpoint is performed on the image.
- the viewpoint composition unit 82 performs warping processing to the display viewpoint for the multi-view corrected color image supplied from the multi-view image decoding unit 81 using the parallax related image of the display viewpoint obtained as a result of the warping processing. Do.
- the viewpoint combining unit 82 supplies the color image of the display viewpoint obtained as a result to the multi-viewpoint image display unit 33 as a multi-viewpoint combined color image.
- the viewpoint combining unit 82 performs the warping process to the display viewpoint for the multi-viewpoint parallax related image with the accuracy corresponding to the viewpoint accuracy parameter based on the parallax accuracy parameter. There is no need to perform high-precision warping processing.
- the viewpoint synthesis unit 82 performs warping processing to the display viewpoint for the multi-viewpoint parallax related image based on the inter-camera distance, the parallax corresponding to the parallax associated value of the multi-viewpoint parallax related image after the warping processing is appropriate If the range is not within the range, the parallax related value can be corrected to a value corresponding to the appropriate range of parallax based on the inter-camera distance.
- FIG. 23 is a block diagram showing a configuration example of the multi-viewpoint image decoding unit 81 of FIG.
- the multi-viewpoint image decoding unit 81 in FIG. 23 includes an SPS decoding unit 101, a PPS decoding unit 102, a slice header decoding unit 103, and a slice decoding unit 104.
- the SPS decoding unit 101 of the multi-viewpoint image decoding unit 81 receives the coded bit stream transmitted from the coding device 50 of FIG. 11 and extracts the SPS of the coded bit stream.
- the SPS decoding unit 101 supplies the extracted SPS and the encoded bit stream other than the SPS to the PPS decoding unit 102.
- the PPS decoding unit 102 extracts the PPS from the coded bit stream other than the SPS supplied from the SPS decoding unit 101.
- the PPS decoding unit 102 supplies the extracted PPS, SPS, and SPS and a coded bit stream other than PPS to the slice header decoding unit 103.
- the slice header decoding unit 103 extracts a slice header from the SPS supplied from the PPS decoding unit 102 and the encoded bit stream other than the PPS.
- the slice header decoding unit 103 determines the inter-camera distance, the parallax related maximum value, and the parallax related minimum contained in the slice header. Hold values or update inter-camera distances, disparity-related maxima, and disparity-related minima maintained based on inter-camera distances, disparity-related maxima, and disparity-encoding minimum results of disparity-related minima Do.
- the slice header decoding unit 103 generates viewpoint generation information from the held inter-camera distance, disparity related maximum value, disparity related minimum value, and disparity accuracy parameter included in PPS, and supplies the information to the viewpoint combining unit 82. Do. Furthermore, the slice header decoding unit 103 supplies, to the slice decoding unit 104, the SPS, the PPS, the slice header, the SPS, the PPS, and the encoded data in slice units, which is a coded bit stream other than the slice header.
- the slice decoding unit 104 performs slice-based encoding in a method corresponding to the encoding method in the slice encoding unit 61 (FIG. 15) based on the SPS and PPS supplied from the slice header decoding unit 103 and the slice header. Decrypt the data.
- the slice header decoding unit 103 supplies the multi-view corrected color image and the multi-view parallax related image obtained as a result of the decoding to the view synthesis unit 82 in FIG.
- FIG. 24 is a flowchart for explaining the multi-viewpoint decoding process of the multi-viewpoint image decoding unit 81 of the decoding device 80 of FIG.
- step S161 of FIG. 24 the SPS decoding unit 101 (FIG. 23) of the multi-viewpoint image decoding unit 81 extracts the SPS of the received encoded bit stream.
- the SPS decoding unit 101 supplies the extracted SPS and the encoded bit stream other than the SPS to the PPS decoding unit 102.
- step S162 the PPS decoding unit 102 extracts the PPS from the coded bit stream other than the SPS supplied from the SPS decoding unit 101.
- the PPS decoding unit 102 supplies the extracted PPS, SPS, and SPS and a coded bit stream other than PPS to the slice header decoding unit 103.
- step S163 the slice header decoding unit 103 supplies the parallax accuracy parameter included in the PPS supplied from the PPS decoding unit 102 to the viewpoint synthesis unit 82 as a part of the viewpoint generation information.
- step S164 the slice header decoding unit 103 determines whether the transmission flag included in the PPS from the PPS decoding unit 102 is “1” indicating the presence of transmission.
- the processing of the subsequent steps S165 to S174 is performed in units of slices.
- step S164 If it is determined in step S164 that the transmission flag is "1" representing the presence of transmission, the process proceeds to step S165.
- step S165 the slice header decoding unit 103 determines, from the SPS and the encoded bit stream other than the PPS supplied from the PPS decoding unit 102, the disparity-related maximum value, the disparity-related minimum value, and the inter-camera distance or disparity-related maximum value. Extract a slice header including the value, the disparity related minimum value, and the differential encoding result of the inter-camera distance.
- step S166 the slice header decoding unit 103 determines whether the slice type is intra type. If it is determined in step S166 that the slice type is intra type, the process proceeds to step S167.
- step S167 the slice header decoding unit 103 holds the disparity-related minimum value included in the slice header extracted in step S165, and supplies the minimum value as a part of viewpoint generation information to the viewpoint combining unit 82.
- step S168 the slice header decoding unit 103 holds the disparity-related maximum value included in the slice header extracted in step S165, and supplies it to the viewpoint combining unit 82 as a part of viewpoint generation information.
- step S169 the slice header decoding unit 103 holds the inter-camera distance included in the slice header extracted in step S165, and supplies the inter-camera distance as a part of viewpoint generation information to the viewpoint combining unit 82. Then, the process proceeds to step S175.
- step S166 determines whether the slice type is the intra type, that is, if the slice type is the inter type. If it is determined in step S166 that the slice type is not the intra type, that is, if the slice type is the inter type, the process proceeds to step S170.
- step S170 the slice header decoding unit 103 adds the difference encoding result of the parallax-related minimum value included in the slice header extracted in step S165 to the held parallax-related minimum value.
- the slice header decoding unit 103 supplies the disparity related minimum value restored by the addition to the viewpoint combining unit 82 as a part of the viewpoint generation information.
- step S171 the slice header decoding unit 103 adds the difference encoding result of the disparity related maximum value included in the slice header extracted in step S165 to the retained disparity related maximum value.
- the slice header decoding unit 103 supplies the disparity related maximum value restored by the addition to the viewpoint combining unit 82 as a part of the viewpoint generation information.
- step S172 the slice header decoding unit 103 adds the inter-camera distance difference encoding result included in the slice header extracted in step S165 to the held inter-camera distance.
- the slice header decoding unit 103 supplies the inter-camera distance restored by the addition to the viewpoint synthesis unit 82 as part of the viewpoint generation information. Then, the process proceeds to step S175.
- step S164 determines whether the transmission flag is not "1" representing the presence of transmission, that is, if the transmission flag is "0" indicating the absence of transmission. If it is determined in step S164 that the transmission flag is not "1" representing the presence of transmission, that is, if the transmission flag is "0" indicating the absence of transmission, the process proceeds to step S173.
- the slice header decoding unit 103 determines the disparity related maximum value, disparity related minimum value, inter-camera distance, and disparity related maximum value from the SPS and the encoded bit stream other than PPS supplied from the PPS decoding unit 102. Extract slice headers that do not include differential encoding results of values, disparity-related minimum values, and inter-camera distances.
- step S174 the slice header decoding unit 103 holds the disparity related maximum value, the disparity related minimum value, and the inter-camera distance, that is, the disparity related maximum value of the immediately preceding slice in the coding order, the disparity related minimum value And the inter-camera distance as the disparity-related maximum value of the slice to be processed, the disparity-related minimum value, and the inter-camera distance, the disparity-related maximum value of the slice to be processed, the disparity-related minimum value, and the inter-camera distance Restore. Then, the slice header decoding unit 103 supplies the restored disparity-related maximum value, disparity-related minimum value, and inter-camera distance to the viewpoint combining unit 82 as a part of viewpoint generation information, and the process proceeds to step S175. .
- step S175 based on the SPS and PPS supplied from the slice header decoding unit 103 and the slice header, the slice decoding unit 104 performs slice processing in a method corresponding to the coding method in the slice coding unit 61 (FIG. 15).
- Decode unit encoded data.
- the slice header decoding unit 103 supplies the multi-view corrected color image and the multi-view parallax related image obtained as a result of the decoding to the view synthesis unit 82 in FIG.
- the decoding device 80 can decode the coded bit stream in which the disparity-related maximum value, the disparity-related minimum value, and the inter-camera distance are arranged in the slice header as information regarding encoding. This makes it possible to decode, for example, a coded bit stream using a disparity-related maximum value, a disparity-related minimum value, and a distance between cameras for encoding.
- the disparity-related maximum value, the disparity-related minimum value, and the inter-camera distance of viewpoint generation information are included in the slice header, but are described in SPS, PPS, SEI, etc. It is also good.
- color image specification information, parallax related image specification information, external parameter information, internal parameter information, number of color image viewpoints, and parallax value are used as the base point in the viewpoint generation information.
- Information for specifying a color image of the viewpoint, the number of viewpoints of the parallax related image, image type information, and the like may be included.
- FIG. 26 shows a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.
- the program can be recorded in advance in a storage unit 808 or a ROM (Read Only Memory) 802 as a recording medium incorporated in the computer.
- ROM Read Only Memory
- the program can be stored (recorded) on the removable medium 811.
- removable media 811 can be provided as so-called package software.
- examples of the removable medium 811 include a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disc, a digital versatile disc (DVD), a magnetic disc, a semiconductor memory, and the like.
- the program may be installed in the computer from the removable media 811 as described above via the drive 810, or may be downloaded to the computer via the communication network or broadcast network and installed in the built-in storage unit 808. That is, for example, the program is wirelessly transferred from the download site to the computer via an artificial satellite for digital satellite broadcasting, or transferred to the computer via a network such as a LAN (Local Area Network) or the Internet. be able to.
- LAN Local Area Network
- the computer incorporates a CPU (Central Processing Unit) 801, and an input / output interface 805 is connected to the CPU 801 via a bus 804.
- a CPU Central Processing Unit
- an input / output interface 805 is connected to the CPU 801 via a bus 804.
- the CPU 801 executes the program stored in the ROM 802 accordingly.
- the CPU 801 loads a program stored in the storage unit 808 into a random access memory (RAM) 803 and executes the program.
- RAM random access memory
- the CPU 801 performs the processing according to the above-described flowchart or the processing performed by the configuration of the above-described block diagram. Then, the CPU 801 causes the processing result to be output from the output unit 807, transmitted from the communication unit 809, or recorded in the storage unit 808, as necessary, for example.
- the input unit 806 is configured of a keyboard, a mouse, a microphone, and the like.
- the output unit 807 is configured of an LCD (Liquid Crystal Display), a speaker, and the like.
- the processing performed by the computer according to the program does not necessarily have to be performed chronologically in the order described as the flowchart. That is, the processing performed by the computer according to the program includes processing executed in parallel or separately (for example, parallel processing or processing by an object).
- the program may be processed by one computer (processor) or may be distributed and processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer for execution.
- the present technology processes when communicating via network media such as satellite, cable TV (the television), the Internet, and mobile phones, or on storage media such as optical, magnetic disks, and flash memory.
- network media such as satellite, cable TV (the television), the Internet, and mobile phones
- storage media such as optical, magnetic disks, and flash memory.
- the present invention can be applied to an image processing apparatus used in
- image processing apparatus described above can be applied to any electronic device.
- the example will be described below.
- FIG. 27 illustrates a schematic configuration of a television to which the present technology is applied.
- the television set 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, and an external interface unit 909.
- the television device 900 includes a control unit 910, a user interface unit 911 and the like.
- the tuner 902 selects a desired channel from the broadcast wave signal received by the antenna 901 and demodulates it, and outputs the obtained encoded bit stream to the demultiplexer 903.
- the demultiplexer 903 extracts a video or audio packet of a program to be viewed from the encoded bit stream, and outputs data of the extracted packet to the decoder 904. Also, the demultiplexer 903 supplies a packet of data such as an EPG (Electronic Program Guide) to the control unit 910. When the scrambling is performed, the scrambling is canceled by a demultiplexer or the like.
- EPG Electronic Program Guide
- the decoder 904 decodes the packet, and outputs the video data generated by the decoding process to the video signal processing unit 905 and the audio data to the audio signal processing unit 907.
- the video signal processing unit 905 performs noise removal, video processing and the like according to user settings on the video data.
- the video signal processing unit 905 generates video data of a program to be displayed on the display unit 906, image data by processing based on an application supplied via a network, and the like. Further, the video signal processing unit 905 generates video data for displaying a menu screen or the like such as item selection, and superimposes the video data on video data of a program.
- the video signal processing unit 905 generates a drive signal based on the video data generated in this manner, and drives the display unit 906.
- the display unit 906 drives a display device (for example, a liquid crystal display element or the like) based on the drive signal from the video signal processing unit 905 to display a video of the program.
- a display device for example, a liquid crystal display element or the like
- the audio signal processing unit 907 performs predetermined processing such as noise removal on the audio data, performs D / A conversion processing and amplification processing of the processed audio data, and supplies the speaker 908 with audio output.
- An external interface unit 909 is an interface for connecting to an external device or a network, and transmits and receives data such as video data and audio data.
- a user interface unit 911 is connected to the control unit 910.
- the user interface unit 911 is configured of an operation switch, a remote control signal reception unit, and the like, and supplies an operation signal according to a user operation to the control unit 910.
- the control unit 910 is configured using a CPU (Central Processing Unit), a memory, and the like.
- the memory stores programs executed by the CPU, various data necessary for the CPU to perform processing, EPG data, data acquired via the network, and the like.
- the program stored in the memory is read and executed by the CPU at a predetermined timing such as when the television device 900 is started.
- the CPU executes the program to control each unit such that the television device 900 operates according to the user operation.
- the television apparatus 900 is provided with a bus 912 for connecting the tuner 902, the demultiplexer 903, the video signal processing unit 905, the audio signal processing unit 907, the external interface unit 909, and the like to the control unit 910.
- the decoder 904 is provided with the function of the image processing apparatus (image processing method) of the present application. Therefore, using a color image of a predetermined viewpoint and a parallax related image, it is possible to generate a color image of a viewpoint other than the viewpoint.
- FIG. 28 illustrates a schematic configuration of a mobile phone to which the present technology is applied.
- the cellular phone 920 includes a communication unit 922, an audio codec 923, a camera unit 926, an image processing unit 927, a multiplexing and separating unit 928, a recording and reproducing unit 929, a display unit 930, and a control unit 931. These are connected to one another via a bus 933.
- an antenna 921 is connected to the communication unit 922, and a speaker 924 and a microphone 925 are connected to the audio codec 923. Further, an operation unit 932 is connected to the control unit 931.
- the mobile phone 920 performs various operations such as transmission and reception of audio signals, transmission and reception of electronic mail and image data, image shooting, data recording, and the like in various modes such as a voice call mode and a data communication mode.
- an audio signal generated by the microphone 925 is converted into audio data and compressed by the audio codec 923 and supplied to the communication unit 922.
- the communication unit 922 performs modulation processing of audio data, frequency conversion processing, and the like to generate a transmission signal. Further, the communication unit 922 supplies a transmission signal to the antenna 921 to transmit it to a base station (not shown). In addition, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921, and supplies the obtained audio data to the audio codec 923.
- the audio codec 923 performs data expansion of audio data and conversion to an analog audio signal, and outputs it to the speaker 924.
- control unit 931 receives the character data input by the operation of operation unit 932, and displays the input character on display unit 930. Further, the control unit 931 generates mail data based on a user instruction or the like in the operation unit 932 and supplies the mail data to the communication unit 922.
- the communication unit 922 performs modulation processing and frequency conversion processing of mail data, and transmits the obtained transmission signal from the antenna 921. Further, the communication unit 922 performs amplification, frequency conversion processing, demodulation processing and the like of the received signal received by the antenna 921 to restore mail data.
- the mail data is supplied to the display unit 930 to display the contents of the mail.
- the portable telephone 920 can also store the received mail data in the storage medium by the recording and reproducing unit 929.
- the storage medium is any rewritable storage medium.
- the storage medium is a removable memory such as a RAM or a semiconductor memory such as a built-in flash memory, a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card.
- the image data generated by the camera unit 926 is supplied to the image processing unit 927.
- the image processing unit 927 performs encoding processing of image data to generate encoded data.
- the demultiplexing unit 928 multiplexes the encoded data generated by the image processing unit 927 and the audio data supplied from the audio codec 923 according to a predetermined method, and supplies the multiplexed data to the communication unit 922.
- the communication unit 922 performs modulation processing and frequency conversion processing of multiplexed data, and transmits the obtained transmission signal from the antenna 921.
- the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921 to restore multiplexed data.
- the multiplexed data is supplied to the demultiplexer 928.
- the demultiplexing unit 928 demultiplexes the multiplexed data, and supplies the encoded data to the image processing unit 927 and the audio data to the audio codec 923.
- the image processing unit 927 decodes encoded data to generate image data.
- the image data is supplied to the display unit 930 to display the received image.
- the audio codec 923 converts audio data into an analog audio signal, supplies the analog audio signal to the speaker 924, and outputs the received audio.
- the image processing unit 927 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, it is possible to transmit information necessary to generate a color image of a viewpoint other than the viewpoint using a color image of a predetermined viewpoint and a parallax related image. In addition, color images of viewpoints other than the viewpoint can be generated using a color image of a predetermined viewpoint and a parallax related image.
- FIG. 29 illustrates a schematic configuration of a recording and reproducing device to which the present technology is applied.
- the recording / reproducing device 940 records, for example, audio data and video data of the received broadcast program on a recording medium, and provides the recorded data to the user at a timing according to the user's instruction.
- the recording / reproducing device 940 can also acquire audio data and video data from another device, for example, and record them on a recording medium. Further, the recording / reproducing device 940 decodes and outputs the audio data and the video data recorded on the recording medium so that the monitor device or the like can perform image display and audio output.
- the recording / reproducing device 940 includes a tuner 941, an external interface unit 942, an encoder 943, a hard disk drive (HDD) unit 944, a disk drive 945, a selector 946, a decoder 947, an on-screen display (OSD) unit 948, and a control unit 949.
- a user interface unit 950 is provided.
- the tuner 941 selects a desired channel from a broadcast signal received by an antenna not shown.
- the tuner 941 demodulates the reception signal of the desired channel, and outputs a coded bit stream obtained to the selector 946.
- the external interface unit 942 is configured by at least one of an IEEE 1394 interface, a network interface unit, a USB interface, a flash memory interface, and the like.
- the external interface unit 942 is an interface for connecting to an external device, a network, a memory card or the like, and receives data such as video data and audio data to be recorded.
- the encoder 943 When the video data and audio data supplied from the external interface unit 942 are not encoded, the encoder 943 performs encoding according to a predetermined method, and outputs the encoded bit stream to the selector 946.
- the HDD unit 944 records content data such as video and audio, various programs and other data on a built-in hard disk, and reads them from the hard disk during reproduction.
- the disk drive 945 records and reproduces signals with respect to the mounted optical disk.
- Optical disks such as DVD disks (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.), Blu-ray disks, etc.
- the selector 946 selects one of the encoded bit streams from the tuner 941 or the encoder 943 and supplies the selected bit stream to either the HDD unit 944 or the disk drive 945 when recording video or audio. Also, the selector 946 supplies the encoded bit stream output from the HDD unit 944 or the disk drive 945 to the decoder 947 at the time of video and audio reproduction.
- the decoder 947 decodes the coded bit stream.
- the decoder 947 supplies the video data generated by performing the decoding process to the OSD unit 948.
- the decoder 947 outputs audio data generated by performing decoding processing.
- the OSD unit 948 generates video data for displaying a menu screen or the like such as item selection, and superimposes the video data on the video data output from the decoder 947 and outputs the video data.
- a user interface unit 950 is connected to the control unit 949.
- the user interface unit 950 includes an operation switch, a remote control signal reception unit, and the like, and supplies an operation signal corresponding to a user operation to the control unit 949.
- the control unit 949 is configured using a CPU, a memory, and the like.
- the memory stores programs executed by the CPU and various data necessary for the CPU to perform processing.
- the program stored in the memory is read and executed by the CPU at a predetermined timing such as when the recording / reproducing device 940 is activated.
- the CPU executes the program to control each unit so that the recording and reproducing apparatus 940 operates according to the user operation.
- the decoder 947 is provided with the function of the image processing apparatus (image processing method) of the present application. Therefore, using a color image of a predetermined viewpoint and a parallax related image, it is possible to generate a color image of a viewpoint other than the viewpoint.
- FIG. 30 illustrates a schematic configuration of an imaging device to which the present technology is applied.
- the imaging device 960 captures an image of an object, displays an image of the object on the display unit, or records the image as image data in a recording medium.
- the imaging device 960 includes an optical block 961, an imaging unit 962, a camera signal processing unit 963, an image data processing unit 964, a display unit 965, an external interface unit 966, a memory unit 967, a media drive 968, an OSD unit 969, and a control unit 970.
- a user interface unit 971 is connected to the control unit 970.
- an image data processing unit 964, an external interface unit 966, a memory unit 967, a media drive 968, an OSD unit 969, a control unit 970 and the like are connected via a bus 972.
- the optical block 961 is configured using a focus lens, an aperture mechanism, and the like.
- the optical block 961 forms an optical image of a subject on the imaging surface of the imaging unit 962.
- the imaging unit 962 is configured using a CCD or CMOS image sensor, generates an electrical signal corresponding to an optical image by photoelectric conversion, and supplies the electrical signal to the camera signal processing unit 963.
- the camera signal processing unit 963 performs various camera signal processing such as knee correction, gamma correction, and color correction on the electric signal supplied from the imaging unit 962.
- the camera signal processing unit 963 supplies the image data processing unit 964 with the image data after camera signal processing.
- the image data processing unit 964 performs encoding processing of the image data supplied from the camera signal processing unit 963.
- the image data processing unit 964 supplies the encoded data generated by performing the encoding process to the external interface unit 966 and the media drive 968. Further, the image data processing unit 964 performs a decoding process of the encoded data supplied from the external interface unit 966 or the media drive 968.
- the image data processing unit 964 supplies the image data generated by performing the decoding process to the display unit 965. Further, the image data processing unit 964 performs a process of supplying image data supplied from the camera signal processing unit 963 to the display unit 965, and superimposes display data acquired from the OSD unit 969 on the image data. Supply to
- the OSD unit 969 generates display data such as a menu screen or an icon including symbols, characters, or figures, and outputs the display data to the image data processing unit 964.
- the external interface unit 966 is formed of, for example, a USB input / output terminal, and is connected to a printer when printing an image.
- a drive is connected to the external interface unit 966 as necessary, removable media such as a magnetic disk and an optical disk are appropriately mounted, and a computer program read from them is installed as necessary.
- the external interface unit 966 has a network interface connected to a predetermined network such as a LAN or the Internet.
- Control unit 970 reads encoded data from memory unit 967 according to an instruction from user interface unit 971, for example, and causes external interface unit 966 to supply the encoded data to another device connected via a network. it can.
- the control unit 970 may obtain encoded data and image data supplied from another device via the network via the external interface unit 966 and supply the same to the image data processing unit 964. it can.
- any removable readable / writable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory is used.
- the recording medium may be of any type as a removable medium, and may be a tape device, a disk, or a memory card. Of course, it may be a noncontact IC card or the like.
- media drive 968 and the recording medium may be integrated, and may be configured by a non-portable storage medium such as, for example, a built-in hard disk drive or a solid state drive (SSD).
- a non-portable storage medium such as, for example, a built-in hard disk drive or a solid state drive (SSD).
- the control unit 970 is configured using a CPU, a memory, and the like.
- the memory stores programs executed by the CPU, various data necessary for the CPU to perform processing, and the like.
- the program stored in the memory is read and executed by the CPU at a predetermined timing such as when the imaging device 960 starts up.
- the CPU executes the program to control each unit so that the imaging device 960 operates according to the user operation.
- the image data processing unit 964 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, it is possible to transmit information necessary to generate a color image of a viewpoint other than the viewpoint using a color image of a predetermined viewpoint and a parallax related image. In addition, color images of viewpoints other than the viewpoint can be generated using a color image of a predetermined viewpoint and a parallax related image.
- the present technology can also have the following configurations.
- An encoding unit that encodes a color image of a viewpoint and a depth image of the viewpoint to generate a bitstream; According to a method of generating a color image of a display viewpoint obtained by performing warping processing using the color image and the depth image, generation of information for viewpoint generation used when generating a color image of the display viewpoint Department,
- An image processing apparatus comprising: a transmission unit that transmits the bit stream generated by the encoding unit and the viewpoint generation information generated by the generation unit.
- the transmission unit transmits the viewpoint generation information as a coding parameter used in coding or decoding.
- the transmission unit transmits a difference between the viewpoint generation information of the depth image and the viewpoint generation information of the depth image located in the coding order before the depth image.
- the transmission unit transmits the viewpoint generation information of the slice when the slice of the bit stream is an intra slice, and transmits the difference of the slice when the slice is an inter slice.
- the image processing apparatus according to claim 1. A setting unit configured to set difference identification information identifying presence or absence of the difference; The image processing apparatus according to (3) or (4), wherein the transmission unit transmits the difference identification information set by the setting unit.
- PPS Picture Parameter Set
- the image processing apparatus according to any one of (1) to (6), wherein the generation unit generates information specifying the color image or information specifying the depth image as the viewpoint generation information.
- the depth image is a depth image consisting of depth values representing the position of the object in the depth direction of each pixel of the color image, or the distance between each pixel of the color image and the pixel of the color image of the base point corresponding to that pixel
- a parallax image consisting of parallax values representing The generation unit generates, as the viewpoint generation information, depth image identification information that identifies whether the depth image is a depth image or a parallax image.
- the image processing apparatus according to claim 1.
- the generation unit When the depth image is the depth image, the generation unit generates, as the viewpoint generation information, information indicating the minimum value and the maximum value of the world coordinate values of the position in the depth direction which can be taken in the depth image.
- the generation unit when the depth image is the parallax image, information indicating a minimum value and a maximum value of parallax on world coordinates that can be taken in the parallax image, and information specifying a color image of the base point,
- the image processing apparatus according to (8), which generates as the viewpoint generation information.
- the generation unit includes a depth minimum value representing a minimum value that can be taken as a pixel value of the depth image, a depth maximum value that represents a maximum value that can be taken as a pixel value of the depth image, and a plurality of the above corresponding to the depth image Generating an inter-imaging position distance which is a distance between imaging positions of a color image as the viewpoint generation information;
- the image processing apparatus according to any one of (1) to (8), wherein the transmission unit transmits the viewpoint generation information as a slice header of the bit stream.
- the image processing device Encoding a color image of a viewpoint and a depth image of the viewpoint to generate a bitstream; According to a method of generating a color image of a display viewpoint obtained by performing warping processing using the color image and the depth image, generation of information for viewpoint generation used when generating a color image of the display viewpoint Step and A transmission step of transmitting the bit stream generated by the processing of the encoding step and the viewpoint generation information generated by the processing of the generation step.
- a method of generating a color image of a display viewpoint obtained by performing warping processing using a color image of a viewpoint, a bit stream obtained as a result of encoding a depth image of the viewpoint, and the color image and the depth image A receiving unit that receives the generated information for viewpoint generation used when generating the color image of the display viewpoint; A decoding unit that decodes the bit stream received by the receiving unit to generate the color image and the depth image; A generating unit that generates a color image of the display viewpoint by performing a warping process using the color image and the depth image generated by the decoding unit and the viewpoint generation information received by the receiving unit.
- an image processing device comprising (13) The image processing apparatus according to (12), wherein the receiving unit receives the viewpoint generation information as a coding parameter used in coding or decoding. (14) The receiving unit receives a difference between the viewpoint generation information of the depth image and the viewpoint generation information of a depth image located in a coding order before the depth image. The generation unit corresponds to the difference using the difference received by the reception unit and the viewpoint generation information of a depth image located in the coding order before the depth image corresponding to the difference. The viewpoint generation information of the depth image to be displayed is restored, and the color image of the display viewpoint is generated by performing the warping process using the viewpoint generation information after the restoration, the color image, and the depth image.
- the image processing apparatus according to (12) or (13).
- the receiving unit receives the viewpoint generation information of the slice when the slice of the bitstream is an intra slice, and receives the difference of the slice when the slice is an inter slice.
- Image processing device (16) The receiving unit receives difference identification information identifying presence or absence of the difference, The image processing apparatus according to (14) or (15), wherein the generation unit restores the viewpoint generation information based on the difference identification information received by the reception unit.
- the image processing apparatus according to (16), wherein the receiving unit receives the difference identification information included in a PPS (Picture Parameter Set) of the bit stream.
- the receiving unit receives, as the viewpoint generation information, information specifying the color image or information specifying the depth image.
- the generation unit identifies the color image and the depth image received by the reception unit based on the viewpoint generation information, performs a warping process on the depth image, and displays the display viewpoint after the warping process.
- the image processing apparatus according to any one of (12) to (17), wherein a color image of the display viewpoint is generated by performing a warping process on the color image using a depth image.
- the depth image is a depth image consisting of depth values representing the position of the object in the depth direction of each pixel of the color image, or the distance between each pixel of the color image and the pixel of the color image of the base point corresponding to that pixel
- a parallax image consisting of parallax values representing The image processing according to any one of (12) to (18), wherein the receiving unit receives, as the viewpoint generation information, information indicating whether the depth image is a depth image or a parallax image. apparatus.
- the receiving unit receives, as the viewpoint generation information, information indicating the minimum value and the maximum value of the world coordinate values of the position in the depth direction which can be taken in the depth image,
- the information representing the minimum value and the maximum value of the parallax on the world coordinates that can be taken in the parallax image and the information specifying the color image of the base point are received as described in (19) Image processing device.
- the receiving unit includes a depth minimum value indicating a minimum value that can be taken as a pixel value of the depth image, a depth maximum value that indicates a maximum value that can be taken as a pixel value of the depth image, and a plurality of the above corresponding to the depth image.
- the image processing apparatus according to any one of (12) to (19), wherein the viewpoint generation information including an imaging position distance which is a distance between imaging positions of a color image is received as a slice header of the bit stream.
- the image processing device According to a method of generating a color image of a display viewpoint obtained by performing warping processing using a color image of a viewpoint, a bit stream obtained as a result of encoding a depth image of the viewpoint, and the color image and the depth image Receiving the generated viewpoint generation information to be used when generating the color image of the display viewpoint; Decoding the bit stream received by the process of the receiving step to generate the color image and the depth image; The color image of the display viewpoint is obtained by performing a warping process using the color image and the depth image generated by the process of the decoding step and the viewpoint generation information received by the process of the receiving step. And an generating step of generating an image processing method.
- Reference Signs List 10 encoding device 14 information generation unit for viewpoint generation, 15 multi-view image encoding unit, 30 decoding device, 31 multi-view image decoding unit, 32 viewpoint combining unit
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Geometry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Computing Systems (AREA)
- Library & Information Science (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Processing Or Creating Images (AREA)
- Studio Devices (AREA)
- Image Generation (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Image Processing (AREA)
Abstract
Description
図25は、視差と奥行きについて説明する図である。
[符号化装置の第1実施の形態の構成例]
図2は、本技術を適用した画像処理装置としての符号化装置の第1実施の形態の構成例を示すブロック図である。
図3は、図2の符号化装置10により生成される符号化ビットストリームのアクセスユニットの構成例を示す図である。
図4は、SEIの一部の記述例を示す図である。
v=Max(0,prec-30)
x=(-1)S・2-(30+v)・n
0<e≦62のとき
v=Max(0,e+prec-31)
x=(-1)S・2e-31・(1+n/2v)
・・・(1)
図5は、図2の符号化装置10の符号化処理を説明するフローチャートである。
図7は、図2の符号化装置10から伝送される符号化ビットストリームを復号する、本技術を適用した画像処理装置としての復号装置の第1実施の形態の構成例を示すブロック図である。
図8は、奥行き画像のワーピング処理を説明する図である。
s(x',y',1)T=A’R'-1[(X,Y,Z)T-(t'x,t'y,t'z)T]
・・・(2)
図9は、図7の復号装置30の復号処理を説明するフローチャートである。この復号処理は、例えば、図2の符号化装置10から符号化ビットストリームが伝送されてきたとき、開始される。
[符号化装置の第2実施の形態の構成例]
図11は、本技術を適用した画像処理装置としての符号化装置の第2実施の形態の構成例を示すブロック図である。
図12は、視点生成用情報の視差関連最大値と視差関連最小値を説明する図である。
図15は、図11の多視点画像符号化部55の構成例を示すブロック図である。
図16は、符号化ビットストリームの構成例を示す図である。
図17は、図16のPPSのシンタックスの例を示す図である。
図18および図19は、スライスヘッダのシンタックスの例を示す図である。
図20は、図11の符号化装置50の符号化処理を説明するフローチャートである。
図22は、図11の符号化装置50から伝送される符号化ビットストリームを復号する、本技術を適用した画像処理装置としての復号装置の第2実施の形態の構成例を示すブロック図である。
図23は、図22の多視点画像復号部81の構成例を示すブロック図である。
図22の復号装置80の復号処理は、図9のステップS52の多視点復号処理を除いて同様に行われるので、以下では、多視点復号処理についてのみ説明する。
[本技術を適用したコンピュータの説明]
次に、上述した一連の処理は、ハードウェアにより行うこともできるし、ソフトウェアにより行うこともできる。一連の処理をソフトウェアによって行う場合には、そのソフトウェアを構成するプログラムが、汎用のコンピュータ等にインストールされる。
[テレビジョン装置の構成例]
図27は、本技術を適用したテレビジョン装置の概略構成を例示している。テレビジョン装置900は、アンテナ901、チューナ902、デマルチプレクサ903、デコーダ904、映像信号処理部905、表示部906、音声信号処理部907、スピーカ908、外部インタフェース部909を有している。さらに、テレビジョン装置900は、制御部910、ユーザインタフェース部911等を有している。
[携帯電話機の構成例]
図28は、本技術を適用した携帯電話機の概略構成を例示している。携帯電話機920は、通信部922、音声コーデック923、カメラ部926、画像処理部927、多重分離部928、記録再生部929、表示部930、制御部931を有している。これらは、バス933を介して互いに接続されている。
[記録再生装置の構成例]
図29は、本技術を適用した記録再生装置の概略構成を例示している。記録再生装置940は、例えば受信した放送番組のオーディオデータとビデオデータを、記録媒体に記録して、その記録されたデータをユーザの指示に応じたタイミングでユーザに提供する。また、記録再生装置940は、例えば他の装置からオーディオデータやビデオデータを取得し、それらを記録媒体に記録させることもできる。さらに、記録再生装置940は、記録媒体に記録されているオーディオデータやビデオデータを復号して出力することで、モニタ装置等において画像表示や音声出力を行うことができるようにする。
[撮像装置の構成例]
図30は、本技術を適用した撮像装置の概略構成を例示している。撮像装置960は、被写体を撮像し、被写体の画像を表示部に表示させたり、それを画像データとして、記録媒体に記録する。
視点のカラー画像と前記視点のデプス画像とを符号化してビットストリームを生成する符号化部と、
前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって、前記表示視点のカラー画像を生成する際に用いる視点生成用情報を生成する生成部と、
前記符号化部により生成された前記ビットストリームと、前記生成部により生成された前記視点生成用情報とを伝送する伝送部と
を備える画像処理装置。
(2)
前記伝送部は、前記視点生成用情報を、符号化または復号の際に用いる符号化パラメータとして伝送する
前記(1)に記載の画像処理装置。
(3)
前記伝送部は、前記デプス画像の前記視点生成用情報と前記デプス画像より符号化順で前に位置するデプス画像の前記視点生成用情報との差分を伝送する
前記(1)または(2)に記載の画像処理装置。
(4)
前記伝送部は、前記ビットストリームのスライスがイントラスライスである場合、前記スライスの前記視点生成用情報を伝送し、前記スライスがインタースライスである場合、前記スライスの前記差分を伝送する
前記(3)に記載の画像処理装置。
(5)
前記差分の有無を識別する差分識別情報を設定する設定部
をさらに備え、
前記伝送部は、前記設定部により設定された前記差分識別情報を伝送する
前記(3)または(4)に記載の画像処理装置。
(6)
前記伝送部は、前記設定部により設定された前記差分識別情報を、前記ビットストリームのPPS(Picture Parameter Set)に含めて伝送する
前記(5)に記載の画像処理装置。
(7)
前記生成部は、前記カラー画像を特定する情報、または、前記デプス画像を特定する情報を、前記視点生成用情報として生成する
前記(1)乃至(6)のいずれかに記載の画像処理装置。
(8)
前記デプス画像は、前記カラー画像の各画素の被写体の奥行方向の位置を表す奥行き値からなる奥行き画像、または、前記カラー画像の各画素と、その画素に対応する基点のカラー画像の画素の距離を表す視差値からなる視差画像であり、
前記生成部は、前記デプス画像が奥行き画像であるか、または、視差画像であるかを識別するデプス画像識別情報を、前記視点生成用情報として生成する
前記(1)乃至(7)のいずれかに記載の画像処理装置。
(9)
前記生成部は、前記デプス画像が前記奥行き画像である場合、前記奥行き画像においてとり得る奥行方向の位置の世界座標値の最小値と最大値を表す情報を、前記視点生成用情報として生成し、
前記生成部は、前記デプス画像が前記視差画像である場合、前記視差画像においてとり得る世界座標上の視差の最小値および最大値を表す情報と、前記基点のカラー画像を特定する情報とを、前記視点生成用情報として生成する
前記(8)に記載の画像処理装置。
(10)
前記生成部は、前記デプス画像の画素値としてとり得る最小値を表すデプス最小値と、前記デプス画像の画素値としてとり得る最大値を表すデプス最大値と、前記デプス画像に対応する複数の前記カラー画像の撮像位置間の距離である撮像位置間距離とを、前記視点生成用情報として生成し、
前記伝送部は、前記視点生成用情報を、前記ビットストリームのスライスヘッダとして伝送する
前記(1)乃至(8)のいずれかに記載の画像処理装置。
(11)
画像処理装置が、
視点のカラー画像と前記視点のデプス画像とを符号化してビットストリームを生成する符号化ステップと、
前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって、前記表示視点のカラー画像を生成する際に用いる視点生成用情報を生成する生成ステップと、
前記符号化ステップの処理により生成された前記ビットストリームと、前記生成ステップの処理により生成された前記視点生成用情報とを伝送する伝送ステップと
を含む画像処理方法。
(12)
視点のカラー画像と前記視点のデプス画像の符号化の結果得られるビットストリームと、前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって生成された、前記表示視点のカラー画像を生成する際に用いる視点生成用情報とを受け取る受け取り部と、
前記受け取り部により受け取られた前記ビットストリームを復号して前記カラー画像と前記デプス画像を生成する復号部と、
前記復号部により生成された前記カラー画像および前記デプス画像と、前記受け取り部により受け取られた前記視点生成用情報とを用いてワーピング処理を行うことにより、前記表示視点のカラー画像を生成する生成部と
を備える画像処理装置。
(13)
前記受け取り部は、前記視点生成用情報を、符号化または復号の際に用いる符号化パラメータとして受け取る
前記(12)に記載の画像処理装置。
(14)
前記受け取り部は、前記デプス画像の前記視点生成用情報と前記デプス画像より符号化順で前に位置するデプス画像の前記視点生成用情報との差分を受け取り、
前記生成部は、前記受け取り部により受け取られた前記差分と、前記差分に対応するデプス画像より前記符号化順で前に位置するデプス画像の前記視点生成用情報とを用いて、前記差分に対応するデプス画像の前記視点生成用情報を復元し、復元後の前記視点生成用情報と、前記カラー画像と、前記デプス画像とを用いてワーピング処理を行うことにより、前記表示視点のカラー画像を生成する
前記(12)または(13)に記載の画像処理装置。
(15)
前記受け取り部は、前記ビットストリームのスライスがイントラスライスである場合、前記スライスの前記視点生成用情報を受け取り、前記スライスがインタースライスである場合、前記スライスの前記差分を受け取る
前記(14)に記載の画像処理装置。
(16)
前記受け取り部は、前記差分の有無を識別する差分識別情報を受け取り、
前記生成部は、前記受け取り部により受け取られた前記差分識別情報に基づいて、前記視点生成用情報を復元する
前記(14)または(15)に記載の画像処理装置。
(17)
前記受け取り部は、前記ビットストリームのPPS(Picture Parameter Set)に含まれる前記差分識別情報を受け取る
前記(16)に記載の画像処理装置。
(18)
前記受け取り部は、前記視点生成用情報として、前記カラー画像を特定する情報、または、前記デプス画像を特定する情報を受け取り、
前記生成部は、前記視点生成用情報に基づいて、前記受け取り部により受け取られた前記カラー画像とデプス画像を特定し、前記デプス画像に対してワーピング処理を行い、ワーピング処理後の前記表示視点のデプス画像を用いて前記カラー画像に対してワーピング処理を行うことにより、前記表示視点のカラー画像を生成する
前記(12)乃至(17)のいずれかに記載の画像処理装置。
(19)
前記デプス画像は、前記カラー画像の各画素の被写体の奥行方向の位置を表す奥行き値からなる奥行き画像、または、前記カラー画像の各画素と、その画素に対応する基点のカラー画像の画素の距離を表す視差値からなる視差画像であり、
前記受け取り部は、前記視点生成用情報として、前記デプス画像が奥行き画像であるか、または、視差画像であるかを表す情報を受け取る
前記(12)乃至(18)のいずれかに記載の画像処理装置。
(20)
前記受け取り部は、前記デプス画像が前記奥行き画像である場合、前記視点生成用情報として、前記奥行き画像においてとり得る奥行方向の位置の世界座標値の最小値と最大値を表す情報を受け取り、前記デプス画像が前記視差画像である場合、前記視差画像においてとり得る世界座標上の視差の最小値および最大値を表す情報と、前記基点のカラー画像を特定する情報とを受け取る
前記(19)に記載の画像処理装置。
(21)
前記受け取り部は、前記デプス画像の画素値としてとり得る最小値を表すデプス最小値と、前記デプス画像の画素値としてとり得る最大値を表すデプス最大値と、前記デプス画像に対応する複数の前記カラー画像の撮像位置間の距離である撮像位置間距離とを含む前記視点生成用情報を、前記ビットストリームのスライスヘッダとして受け取る
前記(12)乃至(19)のいずれかに記載の画像処理装置。
(22)
画像処理装置が、
視点のカラー画像と前記視点のデプス画像の符号化の結果得られるビットストリームと、前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって生成された、前記表示視点のカラー画像を生成する際に用いる視点生成用情報とを受け取る受け取りステップと、
前記受け取りステップの処理により受け取られた前記ビットストリームを復号し、前記カラー画像と前記デプス画像を生成する復号ステップと、
前記復号ステップの処理により生成された前記カラー画像および前記デプス画像と、前記受け取りステップの処理により受け取られた前記視点生成用情報とを用いてワーピング処理を行うことにより、前記表示視点のカラー画像を生成する生成ステップと
を含む画像処理方法。
Claims (22)
- 視点のカラー画像と前記視点のデプス画像とを符号化してビットストリームを生成する符号化部と、
前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって、前記表示視点のカラー画像を生成する際に用いる視点生成用情報を生成する生成部と、
前記符号化部により生成された前記ビットストリームと、前記生成部により生成された前記視点生成用情報とを伝送する伝送部と
を備える画像処理装置。 - 前記伝送部は、前記視点生成用情報を、符号化または復号の際に用いる符号化パラメータとして伝送する
請求項1に記載の画像処理装置。 - 前記伝送部は、前記デプス画像の前記視点生成用情報と前記デプス画像より符号化順で前に位置するデプス画像の前記視点生成用情報との差分を伝送する
請求項1に記載の画像処理装置。 - 前記伝送部は、前記ビットストリームのスライスがイントラスライスである場合、前記スライスの前記視点生成用情報を伝送し、前記スライスがインタースライスである場合、前記スライスの前記差分を伝送する
請求項3に記載の画像処理装置。 - 前記差分の有無を識別する差分識別情報を設定する設定部
をさらに備え、
前記伝送部は、前記設定部により設定された前記差分識別情報を伝送する
請求項3に記載の画像処理装置。 - 前記伝送部は、前記設定部により設定された前記差分識別情報を、前記ビットストリームのPPS(Picture Parameter Set)に含めて伝送する
請求項5に記載の画像処理装置。 - 前記生成部は、前記カラー画像を特定する情報、または、前記デプス画像を特定する情報を、前記視点生成用情報として生成する
請求項1に記載の画像処理装置。 - 前記デプス画像は、前記カラー画像の各画素の被写体の奥行方向の位置を表す奥行き値からなる奥行き画像、または、前記カラー画像の各画素と、その画素に対応する基点のカラー画像の画素の距離を表す視差値からなる視差画像であり、
前記生成部は、前記デプス画像が奥行き画像であるか、または、視差画像であるかを識別するデプス画像識別情報を、前記視点生成用情報として生成する
請求項1に記載の画像処理装置。 - 前記生成部は、前記デプス画像が前記奥行き画像である場合、前記奥行き画像においてとり得る奥行方向の位置の世界座標値の最小値と最大値を表す情報を、前記視点生成用情報として生成し、
前記生成部は、前記デプス画像が前記視差画像である場合、前記視差画像においてとり得る世界座標上の視差の最小値および最大値を表す情報と、前記基点のカラー画像を特定する情報とを、前記視点生成用情報として生成する
請求項8に記載の画像処理装置。 - 前記生成部は、前記デプス画像の画素値としてとり得る最小値を表すデプス最小値と、前記デプス画像の画素値としてとり得る最大値を表すデプス最大値と、前記デプス画像に対応する複数の前記カラー画像の撮像位置間の距離である撮像位置間距離とを、前記視点生成用情報として生成し、
前記伝送部は、前記視点生成用情報を、前記ビットストリームのスライスヘッダとして伝送する
請求項1に記載の画像処理装置。 - 画像処理装置が、
視点のカラー画像と前記視点のデプス画像とを符号化してビットストリームを生成する符号化ステップと、
前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって、前記表示視点のカラー画像を生成する際に用いる視点生成用情報を生成する生成ステップと、
前記符号化ステップの処理により生成された前記ビットストリームと、前記生成ステップの処理により生成された前記視点生成用情報とを伝送する伝送ステップと
を含む画像処理方法。 - 視点のカラー画像と前記視点のデプス画像の符号化の結果得られるビットストリームと、前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって生成された、前記表示視点のカラー画像を生成する際に用いる視点生成用情報とを受け取る受け取り部と、
前記受け取り部により受け取られた前記ビットストリームを復号して前記カラー画像と前記デプス画像を生成する復号部と、
前記復号部により生成された前記カラー画像および前記デプス画像と、前記受け取り部により受け取られた前記視点生成用情報とを用いてワーピング処理を行うことにより、前記表示視点のカラー画像を生成する生成部と
を備える画像処理装置。 - 前記受け取り部は、前記視点生成用情報を、符号化または復号の際に用いる符号化パラメータとして受け取る
請求項12に記載の画像処理装置。 - 前記受け取り部は、前記デプス画像の前記視点生成用情報と前記デプス画像より符号化順で前に位置するデプス画像の前記視点生成用情報との差分を受け取り、
前記生成部は、前記受け取り部により受け取られた前記差分と、前記差分に対応するデプス画像より前記符号化順で前に位置するデプス画像の前記視点生成用情報とを用いて、前記差分に対応するデプス画像の前記視点生成用情報を復元し、復元後の前記視点生成用情報と、前記カラー画像と、前記デプス画像とを用いてワーピング処理を行うことにより、前記表示視点のカラー画像を生成する
請求項12に記載の画像処理装置。 - 前記受け取り部は、前記ビットストリームのスライスがイントラスライスである場合、前記スライスの前記視点生成用情報を受け取り、前記スライスがインタースライスである場合、前記スライスの前記差分を受け取る
請求項14に記載の画像処理装置。 - 前記受け取り部は、前記差分の有無を識別する差分識別情報を受け取り、
前記生成部は、前記受け取り部により受け取られた前記差分識別情報に基づいて、前記視点生成用情報を復元する
請求項14に記載の画像処理装置。 - 前記受け取り部は、前記ビットストリームのPPS(Picture Parameter Set)に含まれる前記差分識別情報を受け取る
請求項16に記載の画像処理装置。 - 前記受け取り部は、前記視点生成用情報として、前記カラー画像を特定する情報、または、前記デプス画像を特定する情報を受け取り、
前記生成部は、前記視点生成用情報に基づいて、前記受け取り部により受け取られた前記カラー画像とデプス画像を特定し、前記デプス画像に対してワーピング処理を行い、ワーピング処理後の前記表示視点のデプス画像を用いて前記カラー画像に対してワーピング処理を行うことにより、前記表示視点のカラー画像を生成する
請求項12に記載の画像処理装置。 - 前記デプス画像は、前記カラー画像の各画素の被写体の奥行方向の位置を表す奥行き値からなる奥行き画像、または、前記カラー画像の各画素と、その画素に対応する基点のカラー画像の画素の距離を表す視差値からなる視差画像であり、
前記受け取り部は、前記視点生成用情報として、前記デプス画像が奥行き画像であるか、または、視差画像であるかを表す情報を受け取る
請求項12に記載の画像処理装置。 - 前記受け取り部は、前記デプス画像が前記奥行き画像である場合、前記視点生成用情報として、前記奥行き画像においてとり得る奥行方向の位置の世界座標値の最小値と最大値を表す情報を受け取り、前記デプス画像が前記視差画像である場合、前記視差画像においてとり得る世界座標上の視差の最小値および最大値を表す情報と、前記基点のカラー画像を特定する情報とを受け取る
請求項19に記載の画像処理装置。 - 前記受け取り部は、前記デプス画像の画素値としてとり得る最小値を表すデプス最小値と、前記デプス画像の画素値としてとり得る最大値を表すデプス最大値と、前記デプス画像に対応する複数の前記カラー画像の撮像位置間の距離である撮像位置間距離とを含む前記視点生成用情報を、前記ビットストリームのスライスヘッダとして受け取る
請求項12に記載の画像処理装置。 - 画像処理装置が、
視点のカラー画像と前記視点のデプス画像の符号化の結果得られるビットストリームと、前記カラー画像と前記デプス画像とを用いてワーピング処理を行うことにより得られる表示視点のカラー画像の生成方法にしたがって生成された、前記表示視点のカラー画像を生成する際に用いる視点生成用情報とを受け取る受け取りステップと、
前記受け取りステップの処理により受け取られた前記ビットストリームを復号し、前記カラー画像と前記デプス画像を生成する復号ステップと、
前記復号ステップの処理により生成された前記カラー画像および前記デプス画像と、前記受け取りステップの処理により受け取られた前記視点生成用情報とを用いてワーピング処理を行うことにより、前記表示視点のカラー画像を生成する生成ステップと
を含む画像処理方法。
Priority Applications (14)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CA2827621A CA2827621C (en) | 2011-03-18 | 2012-03-09 | Image processing apparatus and image processing method |
AU2012232423A AU2012232423A1 (en) | 2011-03-18 | 2012-03-09 | Image processing device and image processing method |
MX2013010423A MX2013010423A (es) | 2011-03-18 | 2012-03-09 | Aparato de procesamiento de imagenes y metodo de procesamiento de imagenes. |
MX2014013390A MX352224B (es) | 2011-03-18 | 2012-03-09 | Aparato de procesamiento de imagenes y metodo de procesamiento de imagenes. |
JP2013505886A JP6057136B2 (ja) | 2011-03-18 | 2012-03-09 | 画像処理装置および画像処理方法 |
US14/003,941 US9615079B2 (en) | 2011-03-18 | 2012-03-09 | Image processing apparatus and image processing method |
EP12760156.5A EP2688280A4 (en) | 2011-03-18 | 2012-03-09 | IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD |
RU2013141792/07A RU2586064C2 (ru) | 2011-03-18 | 2012-03-09 | Устройство обработки изображений и способ обработки изображений |
CN201280012933.1A CN103430529B (zh) | 2011-03-18 | 2012-03-09 | 图像处理设备和图像处理方法 |
BR112013023302A BR112013023302A2 (pt) | 2011-03-18 | 2012-03-09 | aparelho e método de processamento de imagem |
US15/352,348 US9712802B2 (en) | 2011-03-18 | 2016-11-15 | Image processing apparatus and image processing method |
AU2016262741A AU2016262741B2 (en) | 2011-03-18 | 2016-11-24 | Image Processing Apparatus and Image Processing Method |
US15/618,488 US10389997B2 (en) | 2011-03-18 | 2017-06-09 | Image processing apparatus and image processing method |
US15/618,588 US10218958B2 (en) | 2011-03-18 | 2017-06-09 | Image processing apparatus and image processing method |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011061486 | 2011-03-18 | ||
JP2011-061486 | 2011-03-18 | ||
JP2011188815 | 2011-08-31 | ||
JP2011-188815 | 2011-08-31 | ||
JP2012019024 | 2012-01-31 | ||
JP2012-019024 | 2012-01-31 |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/003,941 A-371-Of-International US9615079B2 (en) | 2011-03-18 | 2012-03-09 | Image processing apparatus and image processing method |
US15/352,348 Continuation US9712802B2 (en) | 2011-03-18 | 2016-11-15 | Image processing apparatus and image processing method |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012128070A1 true WO2012128070A1 (ja) | 2012-09-27 |
Family
ID=46879224
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/056084 WO2012128070A1 (ja) | 2011-03-18 | 2012-03-09 | 画像処理装置および画像処理方法 |
Country Status (10)
Country | Link |
---|---|
US (4) | US9615079B2 (ja) |
EP (2) | EP2688280A4 (ja) |
JP (3) | JP6057136B2 (ja) |
CN (5) | CN107277484A (ja) |
AU (2) | AU2012232423A1 (ja) |
BR (1) | BR112013023302A2 (ja) |
CA (1) | CA2827621C (ja) |
MX (2) | MX352224B (ja) |
RU (1) | RU2586064C2 (ja) |
WO (1) | WO2012128070A1 (ja) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017082078A1 (ja) * | 2015-11-11 | 2017-05-18 | ソニー株式会社 | 画像処理装置および画像処理方法 |
WO2017082079A1 (ja) * | 2015-11-11 | 2017-05-18 | ソニー株式会社 | 画像処理装置および画像処理方法 |
CN107071381A (zh) * | 2013-04-12 | 2017-08-18 | 迪士尼企业公司 | 信令传输使用3d视频编码的高效率视频编码扩展的变形图 |
RU2743518C2 (ru) * | 2017-04-11 | 2021-02-19 | Долби Лэборетериз Лайсенсинг Корпорейшн | Восприятия многослойных дополненных развлечений |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPWO2012128069A1 (ja) | 2011-03-18 | 2014-07-24 | ソニー株式会社 | 画像処理装置および画像処理方法 |
US9615079B2 (en) | 2011-03-18 | 2017-04-04 | Sony Corporation | Image processing apparatus and image processing method |
SG10201505802XA (en) | 2011-06-30 | 2015-09-29 | Sony Corp | Image processing device and image processing method |
WO2013031573A1 (ja) | 2011-08-31 | 2013-03-07 | ソニー株式会社 | 符号化装置および符号化方法、復号装置および復号方法 |
WO2013162454A1 (en) * | 2012-04-24 | 2013-10-31 | Telefonaktiebolaget L M Ericsson (Publ) | Identifying a parameter set for decoding a multi-layer video representation |
US10085008B2 (en) * | 2013-09-11 | 2018-09-25 | Sony Corporation | Image processing apparatus and method |
US9824290B2 (en) * | 2015-02-10 | 2017-11-21 | nearmap australia pty ltd. | Corridor capture |
AU2016343499B2 (en) * | 2015-10-19 | 2019-08-29 | Hfi Innovation Inc. | Method and apparatus for decoded picture buffer management in video coding system using intra block copy |
CN106385546A (zh) * | 2016-09-27 | 2017-02-08 | 华南师范大学 | 通过图像处理改善移动电子设备摄像效果的方法和系统 |
JP6419128B2 (ja) * | 2016-10-28 | 2018-11-07 | キヤノン株式会社 | 画像処理装置、画像処理システム、画像処理方法及びプログラム |
US10297087B2 (en) * | 2017-05-31 | 2019-05-21 | Verizon Patent And Licensing Inc. | Methods and systems for generating a merged reality scene based on a virtual object and on a real-world object represented from different vantage points in different video data streams |
US10009640B1 (en) * | 2017-05-31 | 2018-06-26 | Verizon Patent And Licensing Inc. | Methods and systems for using 2D captured imagery of a scene to provide virtual reality content |
US10243088B1 (en) * | 2017-12-21 | 2019-03-26 | Capital One Services, Llc | Transaction card for transferring solar power |
WO2019135365A1 (ja) * | 2018-01-05 | 2019-07-11 | ソニー株式会社 | 画像処理装置、画像処理方法、及び、プログラム |
CN111699469B (zh) * | 2018-03-08 | 2024-05-10 | 三星电子株式会社 | 基于意图的交互式响应方法及其电子设备 |
CN110798674B (zh) * | 2018-08-01 | 2022-04-08 | 中兴通讯股份有限公司 | 图像深度值获取方法、装置、设备、编解码器及存储介质 |
US10638130B1 (en) * | 2019-04-09 | 2020-04-28 | Google Llc | Entropy-inspired directional filtering for image coding |
US10757410B1 (en) * | 2019-07-26 | 2020-08-25 | Google Llc | Spatially adaptive video compression for multiple streams of color and depth |
CN113344822B (zh) * | 2021-06-29 | 2022-11-18 | 展讯通信(上海)有限公司 | 图像降噪方法、装置、终端、存储介质 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2008263528A (ja) * | 2007-04-13 | 2008-10-30 | Univ Nagoya | 画像情報処理方法及び画像情報処理システム |
JP2010157825A (ja) * | 2008-12-26 | 2010-07-15 | Victor Co Of Japan Ltd | 画像符号化装置、画像符号化方法およびそのプログラム |
JP2010267231A (ja) * | 2009-05-18 | 2010-11-25 | Canon Inc | 位置姿勢推定装置及び方法 |
Family Cites Families (49)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH05108804A (ja) * | 1991-10-21 | 1993-04-30 | Nippon Telegr & Teleph Corp <Ntt> | 3次元物体の識別方法及びその実施装置 |
JP3593466B2 (ja) | 1999-01-21 | 2004-11-24 | 日本電信電話株式会社 | 仮想視点画像生成方法およびその装置 |
JP2002058031A (ja) | 2000-08-08 | 2002-02-22 | Nippon Telegr & Teleph Corp <Ntt> | 画像符号化方法及び装置、並びに、画像復号化方法及び装置 |
JP2007166381A (ja) | 2005-12-15 | 2007-06-28 | Univ Of Tokyo | 多視点画像の圧縮符号化方法及び復号化方法 |
JP2008069409A (ja) * | 2006-09-14 | 2008-03-27 | Bridgestone Corp | 高強度高炭素鋼線およびその製造方法 |
CA2663672C (en) * | 2006-09-20 | 2014-08-12 | Nippon Telegraph And Telephone Corporation | Image encoding method and decoding method, apparatuses therefor, programs therefor, and storage media for storing the programs |
WO2008041178A2 (en) | 2006-10-04 | 2008-04-10 | Koninklijke Philips Electronics N.V. | Image enhancement |
CN101222639B (zh) | 2007-01-09 | 2010-04-21 | 华为技术有限公司 | 多视点视频技术中的视间预测方法、编码器和解码器 |
EP2130374A4 (en) * | 2007-03-23 | 2011-03-02 | Lg Electronics Inc | METHOD AND ARRANGEMENT FOR DECODING / CODING A VIDEO SIGNAL |
JP4952657B2 (ja) | 2007-07-19 | 2012-06-13 | 株式会社Jvcケンウッド | 擬似立体画像生成装置、画像符号化装置、画像符号化方法、画像伝送方法、画像復号化装置及び画像復号化方法 |
JP5646994B2 (ja) * | 2007-08-15 | 2014-12-24 | トムソン ライセンシングThomson Licensing | 多視点符号化ビデオにおける領域視差ベクトルを使用したモーションスキップモードのための方法及び装置 |
BRPI0821500A2 (pt) | 2008-01-11 | 2015-06-16 | Thomson Licensing | Codificação de vídeo e profundidade |
US8311353B2 (en) | 2008-01-21 | 2012-11-13 | Telefonaktiebolaget Lm Ericsson (Publ) | Prediction-based image processing |
JP5368482B2 (ja) | 2008-02-05 | 2013-12-18 | トムソン ライセンシング | ビデオ符号化および復号におけるインプリシットなブロック分割のための方法および装置 |
KR101506219B1 (ko) * | 2008-03-25 | 2015-03-27 | 삼성전자주식회사 | 3차원 영상 컨텐츠 제공 방법, 재생 방법, 그 장치 및 그기록매체 |
WO2010021666A1 (en) | 2008-08-20 | 2010-02-25 | Thomson Licensing | Refined depth map |
EP2328337A4 (en) * | 2008-09-02 | 2011-08-10 | Huawei Device Co Ltd | 3D VIDEO COMMUNICATION, TRANSMISSION DEVICE, SYSTEM AND IMAGE RECONSTRUCTION, SYSTEM |
JP5253066B2 (ja) * | 2008-09-24 | 2013-07-31 | キヤノン株式会社 | 位置姿勢計測装置及び方法 |
US8913657B2 (en) * | 2008-10-27 | 2014-12-16 | Lg Electronics Inc. | Virtual view image synthesis method and apparatus |
EP2348732A4 (en) * | 2008-11-10 | 2012-05-09 | Lg Electronics Inc | METHOD AND DEVICE FOR PROCESSING VIDEO SIGNAL BY INTER-VIEW PREDICTION |
US8760495B2 (en) * | 2008-11-18 | 2014-06-24 | Lg Electronics Inc. | Method and apparatus for processing video signal |
WO2010071291A1 (ko) * | 2008-12-18 | 2010-06-24 | (주)엘지전자 | 3차원 영상신호 처리 방법과 이를 구현하기 위한 영상표시 장치 |
CN102265617A (zh) * | 2008-12-26 | 2011-11-30 | 日本胜利株式会社 | 图像编码装置、图像编码方法及其程序、以及图像解码装置、图像解码方法及其程序 |
US20110310982A1 (en) * | 2009-01-12 | 2011-12-22 | Lg Electronics Inc. | Video signal processing method and apparatus using depth information |
KR101630866B1 (ko) * | 2009-01-20 | 2016-06-16 | 코닌클리케 필립스 엔.브이. | 3d 이미지 데이터의 전송 |
EP2384004A4 (en) * | 2009-01-29 | 2012-08-01 | Lg Electronics Inc | METHOD AND APPARATUS FOR PROCESSING VIDEO SIGNALS USING INTRA MODE CONTOUR CODING |
BRPI1008226A2 (pt) | 2009-02-12 | 2019-09-24 | Nippon Telegraph & Telephone | método de codificação de imagem como múltiplas vistas, método de decodificação de imagem com múltiplas vistas, dispositivo de codificação de imagem com múltiplas vistas,dispositivo de codificação de imagem com múltiplas vistas,programa de codificação de imagem como múltiplas vistas, programa de codificação de imagem como múltiplas vistas. |
KR101940023B1 (ko) * | 2009-02-19 | 2019-01-21 | 톰슨 라이센싱 | 3d 비디오 포맷 |
CA2750211C (en) * | 2009-03-19 | 2015-06-16 | Lg Electronics Inc. | Method for processing three dimensional (3d) video signal and digital broadcast receiver for performing the processing method |
US9124874B2 (en) * | 2009-06-05 | 2015-09-01 | Qualcomm Incorporated | Encoding of three-dimensional conversion information with two-dimensional video sequence |
JP5293463B2 (ja) * | 2009-07-09 | 2013-09-18 | ソニー株式会社 | 画像処理装置、画像処理方法およびプログラム |
RU2554465C2 (ru) * | 2009-07-27 | 2015-06-27 | Конинклейке Филипс Электроникс Н.В. | Комбинирование 3d видео и вспомогательных данных |
US20110025830A1 (en) | 2009-07-31 | 2011-02-03 | 3Dmedia Corporation | Methods, systems, and computer-readable storage media for generating stereoscopic content via depth map creation |
US10198792B2 (en) | 2009-10-14 | 2019-02-05 | Dolby Laboratories Licensing Corporation | Method and devices for depth map processing |
US8537200B2 (en) * | 2009-10-23 | 2013-09-17 | Qualcomm Incorporated | Depth map generation techniques for conversion of 2D video data to 3D video data |
JP2012023652A (ja) | 2010-07-16 | 2012-02-02 | Sony Corp | 画像処理装置と画像処理方法 |
JP5505154B2 (ja) | 2010-07-16 | 2014-05-28 | ソニー株式会社 | 画像処理装置と画像処理方法 |
KR102185765B1 (ko) | 2010-08-11 | 2020-12-03 | 지이 비디오 컴프레션, 엘엘씨 | 멀티-뷰 신호 코덱 |
JP6108309B2 (ja) | 2011-02-22 | 2017-04-05 | サン パテント トラスト | 動画像符号化方法、動画像符号化装置、動画像復号方法、および、動画像復号装置 |
US9615079B2 (en) * | 2011-03-18 | 2017-04-04 | Sony Corporation | Image processing apparatus and image processing method |
JPWO2012128069A1 (ja) | 2011-03-18 | 2014-07-24 | ソニー株式会社 | 画像処理装置および画像処理方法 |
WO2012157443A1 (ja) | 2011-05-16 | 2012-11-22 | ソニー株式会社 | 画像処理装置、及び、画像処理方法 |
CN103597833A (zh) | 2011-06-22 | 2014-02-19 | 索尼公司 | 图像处理装置以及方法 |
JP2013034171A (ja) | 2011-06-30 | 2013-02-14 | Sony Corp | 画像処理装置および方法 |
SG10201505802XA (en) | 2011-06-30 | 2015-09-29 | Sony Corp | Image processing device and image processing method |
US9191646B2 (en) | 2011-08-29 | 2015-11-17 | Nokia Technologies Oy | Apparatus, a method and a computer program for video coding and decoding |
WO2013031573A1 (ja) | 2011-08-31 | 2013-03-07 | ソニー株式会社 | 符号化装置および符号化方法、復号装置および復号方法 |
WO2013031575A1 (ja) | 2011-08-31 | 2013-03-07 | ソニー株式会社 | 画像処理装置および画像処理方法 |
JPWO2013115024A1 (ja) | 2012-01-31 | 2015-05-11 | ソニー株式会社 | 画像処理装置、画像処理方法、プログラム、および記録媒体 |
-
2012
- 2012-03-09 US US14/003,941 patent/US9615079B2/en active Active
- 2012-03-09 EP EP12760156.5A patent/EP2688280A4/en not_active Ceased
- 2012-03-09 WO PCT/JP2012/056084 patent/WO2012128070A1/ja active Application Filing
- 2012-03-09 CN CN201710384788.4A patent/CN107277484A/zh active Pending
- 2012-03-09 CN CN201710384783.1A patent/CN107197226B/zh not_active Expired - Fee Related
- 2012-03-09 EP EP17181425.4A patent/EP3267398A3/en not_active Withdrawn
- 2012-03-09 AU AU2012232423A patent/AU2012232423A1/en not_active Abandoned
- 2012-03-09 BR BR112013023302A patent/BR112013023302A2/pt not_active Application Discontinuation
- 2012-03-09 CN CN201280012933.1A patent/CN103430529B/zh not_active Expired - Fee Related
- 2012-03-09 JP JP2013505886A patent/JP6057136B2/ja not_active Expired - Fee Related
- 2012-03-09 RU RU2013141792/07A patent/RU2586064C2/ru not_active IP Right Cessation
- 2012-03-09 CN CN201710385571.5A patent/CN107274475A/zh active Pending
- 2012-03-09 CA CA2827621A patent/CA2827621C/en not_active Expired - Fee Related
- 2012-03-09 MX MX2014013390A patent/MX352224B/es unknown
- 2012-03-09 MX MX2013010423A patent/MX2013010423A/es active IP Right Grant
- 2012-03-09 CN CN201710395370.3A patent/CN107197227B/zh not_active Expired - Fee Related
-
2016
- 2016-11-15 US US15/352,348 patent/US9712802B2/en active Active
- 2016-11-16 JP JP2016223119A patent/JP6365635B2/ja not_active Expired - Fee Related
- 2016-11-24 AU AU2016262741A patent/AU2016262741B2/en not_active Ceased
-
2017
- 2017-06-09 US US15/618,588 patent/US10218958B2/en active Active
- 2017-06-09 US US15/618,488 patent/US10389997B2/en active Active
-
2018
- 2018-06-27 JP JP2018121576A patent/JP2018186528A/ja active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2008263528A (ja) * | 2007-04-13 | 2008-10-30 | Univ Nagoya | 画像情報処理方法及び画像情報処理システム |
JP2010157825A (ja) * | 2008-12-26 | 2010-07-15 | Victor Co Of Japan Ltd | 画像符号化装置、画像符号化方法およびそのプログラム |
JP2010267231A (ja) * | 2009-05-18 | 2010-11-25 | Canon Inc | 位置姿勢推定装置及び方法 |
Non-Patent Citations (3)
Title |
---|
INTERNATIONAL ORGANISATION FOR STANDARDISATION ORGANISATION INTERNATIONALE DE NORMALISATION ISO/IEC JTC1/SC29/WG11 CODING OF MOVING PICTURES AND AUDIO, October 2010 (2010-10-01) |
See also references of EP2688280A4 |
THOMAS WIEGAND; WOO-JIN HAN; BENJAMIN BROSS; JENS-RAINER OHM; GARY J. SULLIVIAN: "WD3: Working Draft3 of High- Efficiency Video Coding", JCTVC-E603_D5, 20 May 2011 (2011-05-20) |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107071381A (zh) * | 2013-04-12 | 2017-08-18 | 迪士尼企业公司 | 信令传输使用3d视频编码的高效率视频编码扩展的变形图 |
CN107071381B (zh) * | 2013-04-12 | 2019-07-09 | 迪士尼企业公司 | 信令传输使用3d视频编码的高效率视频编码扩展的变形图 |
US10757391B2 (en) | 2013-04-12 | 2020-08-25 | Disney Enterprises, Inc. | Signaling warp maps using a high efficiency video coding (HEVC) extension for 3D video coding |
WO2017082078A1 (ja) * | 2015-11-11 | 2017-05-18 | ソニー株式会社 | 画像処理装置および画像処理方法 |
WO2017082079A1 (ja) * | 2015-11-11 | 2017-05-18 | ソニー株式会社 | 画像処理装置および画像処理方法 |
JPWO2017082079A1 (ja) * | 2015-11-11 | 2018-08-30 | ソニー株式会社 | 画像処理装置および画像処理方法 |
US11223812B2 (en) | 2015-11-11 | 2022-01-11 | Sony Corporation | Image processing apparatus and image processing method |
US11290698B2 (en) | 2015-11-11 | 2022-03-29 | Sony Corporation | Image processing apparatus and image processing method |
RU2743518C2 (ru) * | 2017-04-11 | 2021-02-19 | Долби Лэборетериз Лайсенсинг Корпорейшн | Восприятия многослойных дополненных развлечений |
US10991164B2 (en) | 2017-04-11 | 2021-04-27 | Dolby Laboratories Licensing Corporation | Layered augmented entertainment experiences |
US11361520B2 (en) | 2017-04-11 | 2022-06-14 | Dolby Laboratories Licensing Corporation | Layered augmented entertainment experiences |
US11893700B2 (en) | 2017-04-11 | 2024-02-06 | Dolby Laboratories Licensing Corporation | Layered augmented entertainment experiences |
Also Published As
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6365635B2 (ja) | 画像処理装置および画像処理方法 | |
WO2012128069A1 (ja) | 画像処理装置および画像処理方法 | |
US10205927B2 (en) | Encoding device and encoding method, and decoding device and decoding method | |
WO2012114975A1 (ja) | 画像処理装置および画像処理方法 | |
WO2013031575A1 (ja) | 画像処理装置および画像処理方法 | |
WO2013115024A1 (ja) | 画像処理装置および画像処理方法 | |
WO2013031573A1 (ja) | 符号化装置および符号化方法、復号装置および復号方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12760156 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2013505886 Country of ref document: JP Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2827621 Country of ref document: CA |
|
REEP | Request for entry into the european phase |
Ref document number: 2012760156 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2012232423 Country of ref document: AU Date of ref document: 20120309 Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14003941 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 2013141792 Country of ref document: RU Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: MX/A/2013/010423 Country of ref document: MX |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112013023302 Country of ref document: BR |
|
ENP | Entry into the national phase |
Ref document number: 112013023302 Country of ref document: BR Kind code of ref document: A2 Effective date: 20130911 |