JP2013150071A - Encoder, encoding method, program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To encode multi-viewpoint image data and distance information on the multi-viewpoint image data into encoding data of a smaller data amount.SOLUTION: A distance image showing an estimated distance from an imaging section for imaging an input image to a subject of each pixel is acquired on each input image. When view points of the plurality of imaging sections are set in a virtual space in the same arrangement as the reality, and the respective pixels of the distance image corresponding to the respective imaging sections are arranged in a position detached from the view point of the imaging section by a distance which the pixel shows, reliability on the respective imaging sections is set so that reliability becomes higher as a distance difference between the distance from the view points of the imaging sections to the corresponding distance image and a distance from the distance image corresponding to the other imaging section which partially shares a visual field with the imaging section to the view point of the imaging section is smaller. The imaging section in which higher reliability is set is selected. The input image and the distance image, which correspond to the selected imaging section, are encoded.

Description

  The present invention relates to an encoding technique for images and distance information taken from a plurality of viewpoints, and more particularly to an encoding technique for images taken by a multi-view camera.

  In recent years, 3D content has been actively used mainly in the movie industry. Development of multi-view shooting technology and multi-view display technology is progressing in search of a higher sense of reality. Also in the imaging apparatus, a stereo camera is developed as a two-viewpoint photographing apparatus, and a Plenoptic camera, a multi-view camera, a camera array system, and the like are developed as a multi-viewpoint photographing apparatus.

  However, as the number of viewpoints increases, the amount of image data that must be processed also increases. Therefore, there is a need for a format and encoding method that can efficiently handle multi-viewpoint image data. As a standard format for multi-viewpoint images, Multi Picture Format was established by CIPA in 2009 for still images. Regarding moving images, in 2009, in collaboration with ISO / IEC and ITU-T, MVC (Multiview Video Coding) in which H.264 / MPEG-4 is expanded to multi-view video has been standardized. Furthermore, standardization work of MPEG-3DVC (3D Video Coding) is currently in progress. MPEG-3DVC is a method for encoding depth information together with multi-viewpoint images.

  In order to realize higher encoding efficiency, a technique for encoding redundancy in the viewing direction using depth information has also been proposed. For example, a technique for improving the encoding efficiency of a multi-view video by adaptively switching between motion compensation prediction, parallax compensation prediction, depth prediction, and intra-screen prediction has been proposed (Patent Document 1).

JP 2008-271217 A

  However, the above method efficiently encodes image data (and depth information) for each viewpoint of a multi-viewpoint image, and completely solves the problem that the amount of image data increases as the number of viewpoints increases. It is not possible.

  An object of the present invention is to encode multi-view image data and distance information about the multi-view image data into encoded data having a smaller data amount.

In order to achieve the object of the present invention, for example, an encoding apparatus of the present invention comprises the following arrangement. That is,
First acquisition means for acquiring each captured image captured by a plurality of imaging units as an input image;
Second acquisition means for acquiring, for each of the input images, a distance image indicating an estimated distance from an imaging unit that has captured the input image to the subject of each pixel;
In the virtual space, the viewpoints of the plurality of imaging units are set in the same arrangement as in reality, and each pixel of the distance image corresponding to each imaging unit is set at a position separated from the viewpoint of the imaging unit by the distance indicated by the pixel. When arranged, the distance from the viewpoint of each imaging unit to the corresponding distance image, the distance from the distance image corresponding to another imaging unit partially sharing the field of view with the imaging unit, and the viewpoint of the imaging unit, Setting means for setting the reliability for each imaging unit so that the smaller the distance difference is, the higher the reliability is;
Selection means for selecting an imaging unit in which a higher reliability is set;
Encoding means for encoding the input image corresponding to the imaging unit selected by the selection means and the distance image;
It is characterized by providing.

  The multi-view image data and the distance information about the multi-view image data can be encoded into encoded data having a smaller data amount.

It is the figure which showed an example of the encoding apparatus provided with the several imaging part. 1 is a block diagram illustrating an internal configuration of a coding apparatus according to Embodiment 1. FIG. 1 is a diagram illustrating an internal configuration of imaging units 101 to 109 according to Example 1. FIG. 3 is a block diagram illustrating a functional configuration example of an image processing unit 212 according to Embodiment 1. FIG. 4 is a flowchart of processing performed by a reliability calculation unit 403 according to the first embodiment. It is a figure for demonstrating the process which the reliability calculation part 403 performs. It is a figure for demonstrating the process which the reliability calculation part 403 performs. 3 is a flowchart of processing according to the first embodiment.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

  In the first embodiment, it is assumed that a multi-viewpoint image captured by a multi-view camera is encoded. Since the data amount of the multi-viewpoint image is large, it is required to efficiently reduce the data amount of the multi-viewpoint image by encoding. In the first embodiment, attention is paid to the fact that the number of viewpoints required differs depending on the use case. For example, when performing image refocus processing, it is desirable to use images for each viewpoint. On the other hand, the background blurring process using distance information can be performed using an image from one viewpoint and distance information. When use cases are determined in this way, images and distance information for a number of viewpoints suitable for the use cases may be encoded. In this case, since the number of images and distance information to be stored can be reduced, a multi-view image captured by a multi-view camera can be encoded into encoded data with a smaller data amount.

  Specifically, the above-described processing can be performed as follows. First, images captured by the respective imaging units of the multi-lens camera are acquired. Then, distance information about each image is acquired. This distance information may be the distance from the imaging unit to the subject for each pixel of each image. This distance information may be acquired from the imaging unit, or may be calculated by comparing each image as will be described later. Then, the number of images according to the user's instruction is selected, and the selected image and the distance information about the image are encoded.

  When selecting one or more viewpoints from a multi-viewpoint image, what is the criterion for selection is a problem. In this embodiment, images are selected according to the reliability of distance information for each image. Since it is not easy to accurately determine the distance information, the accuracy of the distance information may be different for each viewpoint. Therefore, the result of image processing (for example, background blurring processing) using distance information may differ depending on which viewpoint information is used. It is considered that a better image processing result can be obtained by using distance information with higher accuracy. Therefore, in this embodiment, the reliability (accuracy) of the distance information for each image is obtained, and the distance information with higher reliability and the corresponding image are encoded.

  FIG. 1 illustrates an imaging apparatus that captures an image encoded by the encoding apparatus according to the first embodiment. The imaging device includes a plurality of imaging units. The imaging device can be, for example, a multi-view camera. In the first embodiment, the imaging apparatus further encodes the captured image. In other words, this imaging apparatus operates as the encoding apparatus according to the first embodiment. However, the image encoding may be performed by a device other than the imaging device. That is, the encoding process according to the present embodiment can be realized by a computer including a processor such as the CPU 201 and a memory such as the RAM 202. For example, the processing according to the present embodiment can be realized by reading a computer program stored in a computer-readable storage medium such as the ROM 203 into the RAM 202 and operating the CPU 201 according to the computer program. However, the encoding process according to the present embodiment may be performed by an encoding apparatus including dedicated hardware that implements the processing units illustrated in FIGS. 2 and 4.

  In FIG. 1, reference numeral 100 denotes a casing of the encoding device. The housing 100 includes nine image capturing units 101 to 109 that perform image capturing and a shooting button 110. The nine imaging units 101 to 109 are equally arranged on the square lattice. In the present embodiment, the imaging units 101 to 109 capture a 24-bit color image (captured image).

  When the user presses the shooting button 110, the imaging units 101 to 109 receive light from the subject with a sensor (imaging device) and A / D convert the received signal. In this way, a color image (digital data) is simultaneously acquired by each of the imaging units 101 to 109. With such a multi-eye imaging device (encoding device), it is possible to obtain a color image group obtained by imaging the same subject from a plurality of viewpoints.

  Although the encoding apparatus according to the present embodiment includes nine image capturing units, the number of image capturing units is not limited to nine. That is, the processing of the present embodiment can be applied to an image obtained by an imaging device including a plurality of imaging units. By providing three or more imaging units, an image can be selected according to the reliability as will be described later. In FIG. 1, the imaging units 101 to 109 are evenly arranged on a square lattice. However, the arrangement of the imaging unit is arbitrary. For example, the respective imaging units may be arranged radially or linearly, or may be arranged at random.

  FIG. 2 is a block diagram illustrating an example of an internal configuration of the encoding apparatus according to the present embodiment. FIG. 2 shows a main configuration of the encoding apparatus according to the present embodiment, but the encoding apparatus according to the present embodiment may have other configurations or may not have a part of the configuration. May be.

  Reference numeral 201 denotes a central processing unit (CPU) that comprehensively controls each unit described below. A RAM 202 functions as a main memory or work area of the CPU 201. A ROM 203 stores a control program executed by the CPU 201 and the like. A bus 204 serves as a transfer path for various data. For example, digital data acquired by the imaging units 101 to 109 is sent to a predetermined processing unit via the bus 204.

  Reference numeral 205 denotes an operation unit that receives user instructions. A display unit 206 displays captured images and characters. The display unit 206 may have a touch screen function. In this case, the display unit 206 can receive a user instruction input via a touch screen instead of the operation unit 205. Reference numeral 207 denotes a display control unit, which controls display of captured images and characters displayed on the display unit 206.

  Reference numeral 208 denotes an imaging control unit that controls the imaging units 101 to 109 based on an instruction from the CPU 201. For example, the imaging control unit 208 can perform control such as focusing, opening / closing the shutter, and adjusting the aperture. A digital signal processing unit 209 performs various image processing such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 204. An encoder unit 210 performs a process of converting digital data into a predetermined file format. Details of the encoder unit 210 will be described later.

  Reference numeral 211 denotes an external memory control unit. The external memory control unit 211 is an interface for connecting a PC and other media (for example, hard disk, memory card, CF card, SD card, USB memory). Reference numeral 212 denotes an image processing unit. The image processing unit 212 calculates the distance information and the reliability of the distance information for the color image group acquired by the imaging units 101 to 109 or the color image group output from the digital signal processing unit 209. Then, the image processing unit 212 selects a color image to be encoded based on the calculated reliability. Details of the image processing unit 212 will be described later.

  FIG. 3 shows an internal configuration of the imaging units 101 to 109. The imaging units 101 to 109 include lenses 301 to 303, a diaphragm 304, a shutter 305, an optical low-pass filter 306, an iR cut filter 307, a color filter 308, a sensor 309, and an A / D conversion unit 310. FIG. 3 illustrates the main configuration of the imaging units 101 to 109, but the imaging units 101 to 109 may have other configurations or may not have some configurations.

  The lenses 301 to 303 are a zoom lens 301, a focus lens 302, and a shake correction lens 303, respectively. The sensor 309 is a sensor (imaging device) such as a CMOS or a CCD, for example. When the sensor 309 detects the amount of light from the subject, the detected amount of light is converted into a digital value by the A / D converter 310 and output to the bus 204 as digital data.

  The image processing unit 212 according to the present embodiment will be described with reference to FIG. As illustrated in FIG. 4, the image processing unit 212 includes an image acquisition unit 401, a distance information estimation unit 402, a reliability calculation unit 403, and a viewpoint selection unit 404. Hereinafter, each of these units will be described. The image acquisition unit 401 acquires color images acquired by the imaging units 101 to 109 (first acquisition). Alternatively, the digital signal processing unit 209 may perform image processing on the color image acquired by each of the imaging units 101 to 109, and the image acquisition unit 401 may acquire an image after image processing from the digital signal processing unit 209. Good. Thus, the image acquisition unit 401 acquires image groups from nine viewpoints, each corresponding to the imaging units 101 to 109. Hereinafter, the image groups from nine viewpoints acquired by the image acquisition unit 401 are referred to as input images of the imaging units 101 to 109, respectively. The image acquisition unit 401 sends an input image group from nine viewpoints to the distance information estimation unit 402.

  The distance information estimation unit 402 estimates distance information representing the distance from the camera to the subject for each of the input image groups from nine viewpoints (second acquisition). This distance information indicates the estimated distance from the imaging unit that captured the input image to the subject of each pixel for each pixel. That is, this distance information can be recorded as a distance image. Then, the distance information estimation unit 402 sends the input image group from nine viewpoints and the corresponding distance information to the reliability calculation unit 403.

  The distance information estimation method according to the present embodiment will be briefly described. First, the distance information estimation unit 402 applies a smoothing filter to each input image. In this embodiment, the distance information estimation unit 402 uses an edge holding type smoothing filter. Hereinafter, images obtained by performing filter processing on input images of the imaging units 101 to 109 are referred to as filter images of the imaging units 101 to 109, respectively.

  Next, the distance information estimation unit 402 divides the filter image into a plurality of small regions. Specifically, the distance information estimation unit 402 can perform processing as follows. First, the distance information estimation unit 402 assigns pixels to separate small areas one by one. Thereafter, the distance information estimation unit 402 integrates a plurality of small regions. Specifically, the distance information estimation unit 402 selects one small region and calculates an average color for the selected small region. For example, the average of the pixel values of the pixels constituting the selected small area can be calculated as the average color. Then, the distance information estimation unit 402 compares the average color of the selected small area with the average color of the neighboring small areas, and integrates these small areas when the color difference is equal to or smaller than a predetermined value. However, at this time, the size (number of pixels) of the small area after integration is set within a certain range.

  In this embodiment, it is assumed that the small areas are integrated when the color difference is 6 or less. Also, the color difference is the square root of the sum of squares of the pixel value difference for each of RGB. However, the color difference may be obtained by using another method. For example, a color value in the L * a * b * color system may be calculated, and the color difference may be calculated by the same method. In this embodiment, the integration is performed so that the size of the small area is 100 to 1600 pixels.

  After performing region segmentation for each of the filter images from nine viewpoints, the distance information estimation unit 402 calculates an initial amount of parallax for each small region of each filter image. Specifically, the distance information estimation unit 402 can perform processing as follows. First, the distance information estimation unit 402 selects the viewpoint of the filter image that is the target of initial parallax amount calculation as the reference viewpoint. Here, it is assumed that the distance information estimation unit 402 selects the imaging unit 105 as a reference viewpoint.

  Next, the distance information estimation unit 402 selects a viewpoint around the reference viewpoint as a reference viewpoint. In this embodiment, the upper, lower, left, and right imaging units of the reference viewpoint are selected as reference viewpoints. That is, the distance information estimation unit 402 selects the imaging units 102, 104, 106, and 108 as reference viewpoints. If the distance information estimation unit 402 has selected the imaging unit 107 as a reference viewpoint, the imaging units 104 and 108 are selected as reference viewpoints. When the imaging unit 108 is selected as the reference viewpoint, the imaging units 105, 107, and 109 are selected as reference viewpoints. In this embodiment, the distance information estimation unit 402 thus selects the imaging unit adjacent to the imaging unit that is the reference viewpoint as the reference viewpoint.

  The distance information estimation unit 402 compares each small region included in the reference viewpoint filter image with the reference viewpoint filter image. In this way, the distance information estimation unit 402 calculates the initial amount of parallax for each small region included in the reference viewpoint filter image. Specifically, the distance information estimation unit 402 can sequentially select each small region included in the reference viewpoint filter image, and perform the following processing on the selected small region.

  First, the distance information estimation unit 402 sets an appropriate amount of parallax, and determines a small region corresponding to the selected small region in the reference viewpoint filter image. Here, the position obtained by shifting the position of the selected small area in the filter image of the reference viewpoint by the set amount of parallax is the position of the small area corresponding to the selected small area. Then, the distance information estimation unit 402 calculates, for each pixel, a color difference between the pixel in the selected small area and the corresponding small area pixel. Further, the distance information estimation unit 402 records the number of pixels having the mode value with reference to the calculated color difference histogram. The distance information estimation unit 402 repeatedly records the number of pixels having the mode value while changing the setting value of the parallax amount. Then, the parallax amount with the largest number of pixels having the mode value is selected as the initial parallax amount.

  In this embodiment, the parallax amount is set for both the vertical direction and the horizontal direction. Further, when there are a plurality of reference viewpoints, the initial amount of parallax can be calculated using each reference viewpoint. In this case, the distance information estimation unit 402 may calculate the initial parallax amount using only one reference viewpoint. In another example, the distance information estimation unit 402 may use a statistical value of the initial parallax amount calculated using each reference viewpoint, for example, an average value, as the initial parallax amount below.

  Further, the distance information estimation unit 402 may refine the initial parallax amount calculated in this way. For example, neighboring small area groups with similar colors are likely to have similar amounts of parallax. In addition, there is a high possibility that neighboring small area groups having a small difference in parallax amount have similar parallax amounts. Based on these conditions, the distance information estimation unit 402 can adjust the parallax amount repeatedly using the color difference between the small regions and the difference in the parallax amount. Finally, the distance information estimation unit 402 calculates distance information by converting the amount of parallax into distance information. The conversion from the parallax amount to the distance information can be performed with reference to information indicating the focal length of the imaging units 101 to 109, for example.

  With the distance estimation method described above, distance information corresponding to each pixel of each input image can be calculated. The distance information about each of the input images of the imaging units 101 to 109 can be calculated by selecting the respective imaging units 101 to 109 in turn as the reference viewpoint and performing the distance information calculation process as described above. However, the distance estimation method is not limited to the above-described method, and other methods including a known method may be used. In this embodiment, the distance information estimation unit 402 calculates distance information using the input image, but the distance information estimation unit 402 may receive the distance information. For example, the imaging units 101 to 109 may create distance information by measuring the distance to the subject, and the distance information estimation unit 402 may receive this distance information.

  The reliability calculation unit 403 sets the reliability of the corresponding distance information for each of the imaging units 101 to 109. Then, the reliability calculation unit 403 transmits the input image, distance information, and reliability for each of the imaging units 101 to 109 to the viewpoint selection unit 404. A detailed configuration of the reliability calculation unit 403 will be described later. In the following, the reliability of a viewpoint means the reliability of distance information corresponding to the viewpoint.

  The viewpoint selection unit 404 selects a viewpoint to be encoded from the imaging units 101 to 109 according to the reliability. For example, the viewpoint selection unit 404 may select an imaging unit for which higher reliability is set. Here, the viewpoint selection unit 404 may select an imaging unit with the highest reliability as an encoding target. The viewpoint selection unit 404 may select a plurality of imaging units with higher reliability as encoding targets. However, the viewpoint selection unit 404 does not need to select an imaging unit with the highest reliability as an encoding target. For example, the viewpoint selection unit 404 may select a combination having the highest total reliability among the predetermined combinations of imaging units as an encoding target. By using several combinations of imaging units that are more distant from each other as the predetermined combination of imaging units, the encoded data includes more information and accurate information.

The viewpoint selection unit 404 sends an input image and distance information about the imaging unit to be encoded to the encoder unit 210. The encoder unit 210 encodes the received input image and distance information. Encoding can be performed by any method. The encoder unit 210 may further acquire attribute information of the multiview camera and encode it. Examples of multi-camera attribute information include the following.
-Number of imaging units of multi-view camera-Flag indicating whether distance information is encoded-Number of imaging units to be encoded-Position of imaging unit to be encoded-Direction of rotation of imaging unit to be encoded- Focal length of the imaging unit as the encoding target, angle of view of the imaging unit as the encoding target, distance range for the imaging unit as the encoding target

  In the present embodiment, the encoder unit 210 includes a first encoding unit and a second encoding unit. The first encoding unit encodes the received input image. The second encoding unit encodes the received distance information. The first encoding unit and the second encoding unit may perform encoding using different encoding methods. In this embodiment, it is assumed that both the input image and the distance information are encoded by the JPEG method. When distance information is not necessary, only the input image may be encoded. In this case, the attribute information of the multi-view camera may not have a flag indicating whether the distance information is encoded.

  Next, processing according to the present embodiment will be described with reference to the flowchart of FIG. In step S <b> 801, the image acquisition unit 401 acquires input images of the imaging units 101 to 109. In step S802, the distance information estimation unit 402 calculates distance information for each of the input images of the imaging units 101 to 109 as described above. In step S803, the reliability calculation unit 403 calculates the reliability of corresponding distance information for each of the imaging units 101 to 109. The processing in step S803 will be described in detail later. In step S804, the viewpoint selection unit 404 selects the imaging unit to be encoded as described above. In step S805, the encoder unit 210 encodes the input image and distance information for the imaging unit selected as described above.

  Here, the process of step S803 will be described with reference to the flowchart of FIG. In step S <b> 803, the reliability calculation unit 403 sequentially selects the imaging units 101 to 109 as the reference viewpoint (target imaging unit). And the reliability about a reference viewpoint is calculated by performing the process of step S501-S513 mentioned later about the input image of a reference viewpoint. By repeating the processing of steps S501 to S513 for each imaging unit, the reliability for each imaging unit 101 to 109 can be calculated.

  In step S501, the reliability calculation unit 403 selects viewpoints around the reference viewpoint as reference viewpoints (other imaging units). Here, the reference viewpoint partially shares the field of view with the reference viewpoint. That is, the same subject is included in the reference viewpoint input image and the reference viewpoint input image. In this case, the reference viewpoint is considered to partially share the field of view with the reference viewpoint.

  In this embodiment, the upper, lower, left, and right imaging units of the reference viewpoint are selected as reference viewpoints. Here, it is assumed that the imaging unit 105 is selected as the reference viewpoint. In this case, the reliability calculation unit 403 selects the imaging units 102, 104, 106, and 108 as reference viewpoints. If the reliability calculation unit 403 selects the image capturing unit 107 as a reference viewpoint, the image capturing units 104 and 108 are selected as reference viewpoints. When the imaging unit 108 is selected as the reference viewpoint, the imaging units 105, 107, and 109 are selected as reference viewpoints. Thus, in this embodiment, the number of reference viewpoints is 2-4. The reliability calculation unit 403 may select the reference viewpoint selected for calculating the distance information by the distance information estimation unit 402 in order to calculate the reliability in step S501.

  In step S502, the reliability calculation unit 403 determines whether or not processing for all reference viewpoints has been performed. If processing for all reference viewpoints has been performed, processing proceeds to step S513. If all the reference viewpoints have not been processed, the process proceeds to step S503.

  In step S503, the reliability calculation unit 403 selects one unprocessed reference viewpoint. The order of selection at this time is arbitrary. In step S504, the reliability calculation unit 403 determines whether or not processing has been performed on all pixels of the input image of the selected reference viewpoint. If the process for all pixels has been completed, the process returns to step S502. If the process has not been completed for all pixels, the process proceeds to step S505.

  In step S505, the reliability calculation unit 403 selects one unprocessed pixel (target pixel) from the input image of the selected reference viewpoint. The order of selection at this time is arbitrary. For example, pixels can be selected in order from the upper left to the lower right of the input image.

  In step S506, the reliability calculation unit 403 projects the selected pixel on the 3D space (in the virtual space) with reference to the distance information. Hereinafter, the process of step S506 will be described in more detail with reference to FIG.

  FIG. 6A illustrates an arrangement in the 3D space when the imaging unit 105 is a reference viewpoint and the imaging units 104 and 106 are reference viewpoints. As described above, when the imaging unit 105 is the reference viewpoint, the reference viewpoints are 102, 104, 106, and 108, but only the imaging units 104 and 106 are assumed to be reference viewpoints for the sake of simplicity of explanation. The imaging units 104 and 106 are arranged in the 3D space in the same arrangement as in the real space.

  The object 601 is an object obtained by projecting each pixel of the input image of the imaging unit 104 onto the 3D space according to the distance information. Specifically, the object 601 is obtained by disposing each pixel of the input image of the imaging unit 104 in a direction corresponding to each pixel at a position away from the imaging unit 104 by a distance corresponding to each pixel. The distance corresponding to each pixel can be obtained by referring to the distance information. The direction corresponding to each pixel can be determined with reference to, for example, information indicating the angle of view of the imaging unit 104 and the resolution of the input image. Similarly, the object 602 is an object obtained by projecting each pixel of the input image of the imaging unit 105 onto the 3D space according to the distance information. Furthermore, the object 603 is an object obtained by projecting each pixel of the input image of the imaging unit 106 onto the 3D space according to the distance information.

  In step S506, the reliability calculation unit 403 projects the selected pixel in the direction 604 corresponding to the selected pixel. A 3D point 605 indicating the position of the projected pixel is separated from the imaging unit 104 by a distance corresponding to the selected pixel. That is, the 3D point 605 exists on the object 601 described above. If the imaging unit 106 is the reference viewpoint, the reliability calculation unit 403 projects the selected pixel in a direction 611 corresponding to the selected pixel. The 3D point 610 indicating the position of the projected pixel is separated from the imaging unit 106 by a distance corresponding to the selected pixel.

  FIG. 6B shows an arrangement in the 3D space when the imaging unit 108 is a reference viewpoint and the imaging units 107 and 109 are reference viewpoints. Similar to the case of FIG. 6A, the objects 614, 615, and 616 are obtained by projecting the pixels of the input images of the imaging units 107, 108, and 109 onto the 3D space, respectively. In this case, when the imaging unit 107 is the reference viewpoint, the reliability calculation unit 403 projects the selected pixel in a direction 617 corresponding to the selected pixel. A 3D point 618 indicating the position of the projected pixel is separated from the imaging unit 107 by a distance corresponding to the selected pixel. When the imaging unit 109 is the reference viewpoint, the reliability calculation unit 403 projects the selected pixel in a direction 624 corresponding to the selected pixel. The 3D point 623 indicating the position of the projected pixel is separated from the imaging unit 109 by a distance corresponding to the selected pixel.

  In step S507, the reliability calculation unit 403 determines a direction from the reference viewpoint toward the pixel projected in step S506. For example, in FIG. 6A, when the imaging unit 104 is the reference viewpoint, this direction is a direction 607 from the reference viewpoint 105 toward the 3D point 605. When the imaging unit 106 is a reference viewpoint, this direction is a direction 608 from the reference viewpoint 105 toward the 3D point 610. In the case of FIG. 6B, if the imaging unit 107 is the reference viewpoint, this direction is a direction 620 from the reference viewpoint 108 toward the 3D point 618. When the imaging unit 109 is the reference viewpoint, this direction is a direction 621 from the reference viewpoint 108 toward the 3D point 623.

  In step S508, the reliability calculation unit 403 determines whether the image in the direction determined in step S507 is included in the reference viewpoint input image. Specifically, the reliability calculation unit 403 can know the angle of view of the input image at the reference viewpoint by referring to the focal length of the input image at the reference viewpoint. Then, when the direction determined in step S507 is within the angle of view of the reference viewpoint input image, the reliability calculation unit 403 includes the direction image determined in step S507 in the reference viewpoint input image. Can be determined. If included, the process proceeds to step S509. If not included, the process returns to step S504.

  In step S509, the reliability calculation unit 403 identifies the pixel position in the input image of the reference viewpoint corresponding to the direction determined in step S507. The processing in step S507 can be performed with reference to the focal length of the input image at the reference viewpoint, the resolution of the input image, and the like. The pixel position calculated in step S509 may be between pixels constituting the input image.

  In step S510, the reliability calculation unit 403 determines whether or not the pixel selected in step S505 is in the occlusion region of the reference viewpoint. The occlusion area of the reference viewpoint refers to a portion that is hidden from the shadow of the subject from the reference viewpoint. The reliability calculation unit 403 can determine the range of the occlusion region in the reference viewpoint input image. Whether or not the pixel is in the occlusion region can be determined using a known image processing technique, and thus will not be described in detail in this specification.

  The process of step S510 will be described with reference to FIG. In FIG. 7, the reference viewpoint is the imaging unit 105, and the reference viewpoint is the imaging unit 106. The object 701 is an object obtained by projecting each pixel of the input image of the imaging unit 105 onto the 3D space according to the distance information. The object 702 is an object obtained by projecting each pixel of the input image of the imaging unit 106 onto the 3D space according to the distance information. In step S506, the reliability calculation unit 403 projects the selected pixel in the direction 704 corresponding to the selected pixel. A 3D point 706 indicating the position of the projected pixel is separated from the imaging unit 106 by a distance corresponding to the selected pixel. In step S507, the reliability calculation unit 403 determines a direction 703 from the reference viewpoint toward the projected pixel. The 3D point 705 is located in the direction 703 from the reference viewpoint, and the position of the 3D point 705 is separated from the imaging unit 105 by a distance corresponding to the pixel at the pixel position calculated in step S509. In the occlusion area, corresponding 3D points 705 and 706 represent different objects. In this embodiment, in such a case, the distance information about the 3D points 705 and 706 is not used for calculating the reliability.

  If it is determined in step S510 that it is in the occlusion area, the process returns to step S504. If it is determined in step S510 that it is not in the occlusion area, the process proceeds to step S511. By this processing, the reliability can be calculated using pixels outside the occlusion area of the reference viewpoint in the reference viewpoint input image. However, it is not essential to perform the process of step S510. The calculation of the occlusion area may be performed in units of pixels or may be performed in units of small areas. When the calculation is performed in units of small areas, it can be considered that the small area is in the occlusion area when the characteristics of the corresponding areas of the input image of the base viewpoint and the input image of the reference viewpoint are greatly different.

  In this embodiment, when the pixel selected in step S505 is in the occlusion region of the reference viewpoint, the distance information about this pixel is not used for the reliability calculation. However, the reliability calculation unit 403 can be configured so that the distance information related to any one of the occlusion areas is not used for calculation of the reliability. For example, in another embodiment, when the pixel position calculated in step S509 is in the occlusion area of the reference viewpoint, the distance information about this pixel is not used for the reliability calculation.

  In step S511, the reliability calculation unit 403 acquires the distance (second distance) from the reference viewpoint indicated by the distance information for the pixel position specified in step S509. When the pixel position specified in step S509 is between the pixels constituting the input image, the reliability calculation unit 403 can acquire the distance from the reference viewpoint by interpolating the distance information.

  In step S512, the reliability calculation unit 403 calculates a difference between the distance acquired in step S511 and the distance (first distance) from the reference viewpoint for the pixel selected in step S505. Then, the reliability calculation unit 403 stores the calculated difference in a memory (not shown). Thereafter, the processing returns to step S504.

  With reference to FIG. 6 (A), the process of step S511 and step S512 is demonstrated in detail. The 3D points 606 and 609 indicate pixel positions when the pixel at the pixel position calculated in step S509 is projected in the directions 607 and 608 determined in step S507. The distance acquired in step S511 corresponds to the distance between the 3D points 606 and 609 and the imaging unit 105. Further, the distance from the reference viewpoint for the pixel selected in step S <b> 505 corresponds to the distance between the 3D points 605 and 610 and the imaging unit 105. Therefore, the difference calculated in step S512 corresponds to the distances 612 and 613.

  Here, the 3D points 605 and 609 are points on the objects 601 and 603. Objects 601 and 603 are obtained according to distance information corresponding to the imaging units 104 and 106. That is, when the distance information is represented as a distance image, the objects 601 and 603 are arranged by disposing the pixels of the distance image corresponding to the imaging units 104 and 106 away from the imaging unit by the distance indicated by the pixels. can get. As described above, the distance between the 3D points 605 and 609 and the imaging unit 105 corresponds to the distance between the imaging unit 105 and the distance image corresponding to the imaging units 104 and 106 arranged in the virtual space.

  Similarly, 3D points 606 and 610 are points on the object 602. The object 602 is obtained according to distance information corresponding to the imaging unit 105. That is, when the distance information is represented as a distance image, the object 602 is obtained by disposing each pixel of the distance image corresponding to the imaging unit 105 away from the imaging unit by the distance indicated by the pixel. As described above, the distance between the 3D points 606 and 610 and the imaging unit 105 corresponds to the distance between the imaging unit 105 and the distance image corresponding to the imaging unit 105 arranged in the virtual space.

  The 3D point 606 is located on the direction from the imaging unit 105 toward the 3D point 605. As described above, in the present embodiment, the reliability calculation unit calculates the distance difference between the distance images viewed from the reference viewpoint with respect to the portion where the distance images overlap as viewed from the reference viewpoint. Specifically, the distance difference between the distance image corresponding to the imaging unit 105 that is the reference viewpoint and the distance images corresponding to the imaging units 104 and 106 that are the reference viewpoints is calculated. In particular, in the present embodiment, the distance difference is calculated using the distance information about the field of view shared between the base image and the reference image.

  Similarly, in the case of FIG. 6B, 3D points 619 and 626 indicate pixel positions when the pixels at the pixel positions calculated in step S509 are projected in the directions 620 and 621 determined in step S507. The distance acquired in step S511 corresponds to the distance between the 3D points 619 and 626 and the imaging unit 105. Further, the distance from the reference viewpoint for the pixel selected in step S <b> 505 corresponds to the distance between the 3D points 618 and 623 and the imaging unit 105. Therefore, the distance difference calculated in step S512 corresponds to the distances 625 and 626.

  In the example of FIG. 6, the distance difference in FIG. 6B is smaller than that in FIG. A small distance difference indicates that the distance indicated by the distance information for the reference image is closer to the distance indicated by the distance information for the reference image, which indicates that the accuracy of the distance information is high. Therefore, as the distance difference calculated in step S512 is smaller, it can be determined that the distance information about the reference image is more reliable.

  In step S513, the reliability calculation unit 403 calculates the reliability for the reference viewpoint according to the distance difference recorded in the memory in step S512. The reliability calculation unit calculates the reliability of the reference viewpoint such that the reliability increases as the distance difference between the distance image corresponding to the reference viewpoint and the distance image corresponding to the reference viewpoint decreases. In the present embodiment, the reliability calculation unit uses the average value of the distance differences recorded in the memory in step S512 as the reliability. In this case, smaller reliability means higher reliability. However, the reliability may be obtained by other methods, and another statistical value (for example, maximum value) of the distance difference may be obtained as the reliability.

  In the present embodiment, the configuration and processing of each unit have been described on the assumption that images captured by the imaging units 101 to 109 are color images. However, some or all of the images captured by the imaging units 101 to 109 may be monochrome images. In this case, the color filter 308 can be omitted from the imaging units 101 to 109.

  The calculation method of the reliability described in the present embodiment is merely an example, and another method in which the reliability of the reference viewpoint becomes higher as the distance difference between the distance images viewed from each imaging unit or imaging apparatus is smaller. It can also be adopted. For example, in this embodiment, the reliability is calculated using the distance between the 3D point 605 and the 3D point 606. However, instead, the reliability can be calculated similarly using the difference between the distance between the imaging unit 104 and the 3D point 605 and the distance between the imaging unit 105 and the 3D point 606. In addition, for the object (distance image) 601 and the object 602, an average interval may be calculated for a portion overlapping with the imaging unit 105 (or the plane on which the imaging units 101 to 109 are arranged). The reliability can be similarly calculated using this average interval.

  In the first embodiment, the imaging unit to be encoded is selected according to the reliability of the distance information. In the second embodiment, an imaging unit to be encoded is further selected according to the degree of occlusion. When the degree of occlusion is large for a certain input image, this input image is considered to include many subjects that are not shown in other images. In the present embodiment, such an input image is also an encoding target.

  Hereinafter, an encoding apparatus and a processing method thereof according to the second embodiment will be described. The configuration of the encoding apparatus according to the second embodiment is the same as that of the first embodiment, and a description thereof will be omitted. In addition, the configuration of the image processing unit 212 included in the encoding apparatus according to the second embodiment is the same as that of the first embodiment, and thus the description thereof is omitted.

  The process according to the second embodiment is the same as the process according to the first embodiment illustrated in FIGS. 8 and 5, but the processes in step S510, step S513, and step S804 are different from those in the first embodiment.

  In step S510, when it is determined that the pixel selected in step S505 is in the occlusion region of the reference viewpoint, the reliability calculation unit 403 calculates the occlusion amount for the pixel selected in step S505. Then, the process returns to step S504. If it is determined that it is not in the occlusion area, the process proceeds to step S511 as in the first embodiment.

  The occlusion amount will be described with reference to FIG. Each element shown in FIG. 7 has already been described in connection with the first embodiment. In the present embodiment, the reliability calculation unit 403 calculates the distance between the 3D points 705 and 706 as an occlusion amount, and records the calculated occlusion amount in a memory (not shown).

  In step S513, the reliability calculation unit 403 calculates the reliability for the reference viewpoint according to the distance difference recorded in the memory in step S512. In this embodiment, the reliability calculation unit 403 further calculates an occlusion amount (degree of occlusion) for the input image at the reference viewpoint according to the occlusion amount recorded in the memory in step S509. In the present embodiment, the reliability calculation unit calculates the average value of the occlusion amounts recorded in the memory in step S509 as the occlusion amount for the input image at the reference viewpoint. However, another statistical value (for example, maximum value) of the recorded occlusion amount may be obtained as the occlusion amount for the input image at the reference viewpoint.

  In step S804, the viewpoint selection unit 404 selects an imaging unit to be encoded from the imaging units 101 to 109 according to the reliability as in the first embodiment. In the present embodiment, the viewpoint selection unit 404 further selects an imaging unit to be encoded according to the occlusion amount obtained in step S513. Specifically, when the occlusion amount obtained in step S513 is larger than the set threshold value, the corresponding imaging unit can be selected as an encoding target.

  In this embodiment, the viewpoint selection unit 404 selects an imaging unit to be encoded according to the reliability, and then additionally selects an imaging unit to be encoded according to the occlusion amount, as in the first embodiment. . However, other methods may be used as long as the imaging unit to be encoded is determined according to the reliability and the occlusion amount. For example, the viewpoint selection unit 404 may select an imaging unit to be encoded so that the number of imaging units to be encoded is constant.

  In this example, the degree of occlusion was calculated according to the distance between 3D points as described above. The distance between the 3D points corresponds to the error of the distance information for each input image, and the greater the error, the greater the degree of occlusion. However, the degree of occlusion need not be calculated as in this embodiment. For example, as the degree of occlusion, an index representing the size of the occlusion area and the size of the background object hidden behind the foreground object may be used. That is, an imaging unit that captures an input image having a larger occlusion area may be selected as an imaging unit to be encoded. Here, the size of the occlusion area may indicate the size of an area that is not imaged by any of the other imaging units, or may indicate the size of an area that is not captured by any of the other imaging units. . For example, the size (number of pixels) of the area specified as the occlusion area can be used as an index.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that implements the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (8)

First acquisition means for acquiring each captured image captured by a plurality of imaging units as an input image;
Second acquisition means for acquiring, for each of the input images, a distance image indicating an estimated distance from an imaging unit that has captured the input image to the subject of each pixel;
In the virtual space, the viewpoints of the plurality of imaging units are set in the same arrangement as in reality, and each pixel of the distance image corresponding to each imaging unit is set at a position separated from the viewpoint of the imaging unit by the distance indicated by the pixel. When arranged, the distance from the viewpoint of each imaging unit to the corresponding distance image, the distance from the distance image corresponding to another imaging unit partially sharing the field of view with the imaging unit, and the viewpoint of the imaging unit, Setting means for setting the reliability for each imaging unit so that the smaller the distance difference is, the higher the reliability is;
Selection means for selecting an imaging unit in which a higher reliability is set;
Encoding means for encoding the input image corresponding to the imaging unit selected by the selection means and the distance image;
An encoding device comprising: The setting means includes
Means for acquiring, as a first distance, a distance between a viewpoint of the target imaging unit that sets the reliability in the virtual space and a target pixel of a distance image corresponding to the other imaging unit;
Means for determining a direction from the viewpoint of the target imaging unit to the target pixel in the virtual space;
Means for identifying a pixel in the input image captured by the imaging unit of interest corresponding to the determined direction;
Means for referring to the distance image and acquiring a distance between a pixel of the distance image corresponding to the identified pixel and a viewpoint of the imaging unit of interest as a second distance;
Means for calculating the distance difference using a difference between the first distance and the second distance;
The encoding device according to claim 1, comprising:   The said setting means is further provided with a means to specify an occlusion area | region, The said focused pixel is selected from the outside of the occlusion area | region of the said focused imaging part among the distance images corresponding to the said other imaging part. 2. The encoding device according to 2.   The encoding apparatus according to any one of claims 1 to 3, wherein the setting unit selects, as the other imaging unit, an imaging unit adjacent to an imaging unit that sets the reliability.   5. The encoding apparatus according to claim 1, wherein the selection unit further selects an imaging unit that has captured an input image having a larger occlusion area. 6. An encoding method performed by an encoding device,
A first acquisition unit that acquires, as an input image, each captured image captured by a plurality of imaging units;
A second acquisition step in which a second acquisition unit acquires, for each of the input images, a distance image indicating an estimated distance from an imaging unit that has captured the input image to a subject of each pixel;
The setting means sets the viewpoints of the plurality of imaging units in the virtual space in the same arrangement as in reality, and sets each pixel of the distance image corresponding to each imaging unit from the viewpoint of the imaging unit by the distance indicated by the pixel. When arranged at a separated position, the distance from the viewpoint of each imaging unit to the corresponding distance image, and the distance image corresponding to another imaging unit partially sharing the field of view with the imaging unit to the viewpoint of the imaging unit A setting step for setting the reliability for each imaging unit so that the reliability is higher as the distance difference is smaller,
A selection step in which the selection means selects an imaging unit in which the greater reliability is set;
An encoding step for encoding the input image corresponding to the imaging unit selected in the selection step and the distance image;
An encoding method comprising:   A computer program for causing a computer to function as each unit of the encoding device according to any one of claims 1 to 5.   A computer-readable storage medium storing the computer program according to claim 7.

Images