JP2017130953A - Encoding device, imaging device, encoding method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem in which: when encoding a multi-viewpoint image obtained by photographing a subject from a plurality of viewpoints, a prediction error in inter-viewpoint prediction increases, which reduces encoding efficiency.SOLUTION: Acquisition means acquires a multi-viewpoint image including plural images obtained by photographing a subject from a plurality of viewpoints by using photographing parameters set for respective viewpoints. In the case of encoding an encoding target image out of the images, encoding means encodes the encoding target image by using as a reference viewpoint an image within a predetermined range whose parameters are based on the photographing parameters in the encoding target image, out of the images, and whose parallax from a view point of having imaging the encoding target image is the smallest, out of the images within the predetermined range.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for encoding images taken from a plurality of viewpoints.

  Conventionally, there is a method of generating an image at an arbitrary focus distance, depth of field, and viewpoint using a multi-viewpoint image obtained by photographing a subject from a plurality of viewpoints. The data capacity can be reduced by compressing and encoding the captured multi-viewpoint image (hereinafter referred to as encoding). Data encoded in this way can be recorded on a recording medium or transferred to a server via a network.

  A method of using inter-view prediction encoding (hereinafter referred to as inter-view prediction) when encoding a captured multi-view image has been proposed (Patent Document 1). When each captured multi-viewpoint image is an image of a close viewpoint, there is a correlation between these images. Inter-viewpoint prediction uses the correlation between images. In Patent Document 1, a subject is photographed from a plurality of viewpoints using a plurality of cameras, and a multi-viewpoint image is acquired. Then, one of the plurality of viewpoints is set as the reference viewpoint.

  When encoding an image obtained by photographing from a viewpoint set as a reference viewpoint, an image photographed at a past time at the same viewpoint as the reference viewpoint (hereinafter, the same viewpoint image) is used as a reference image. On the other hand, when encoding an image obtained by photographing from a viewpoint other than the reference viewpoint, the same viewpoint image and an image photographed at an adjacent viewpoint close to the reference viewpoint (hereinafter referred to as an adjacent viewpoint image) are used as reference images. .

JP 9-0261653 A

  In the above-mentioned Patent Document 1, inter-viewpoint prediction uses the same viewpoint image and the adjacent viewpoint image as reference image candidates. Therefore, even if the reference image is the same viewpoint image or an adjacent image, if the shooting conditions are different, the prediction error in the inter-viewpoint prediction becomes large, and the encoding efficiency decreases. In addition, when an image that is blacked out or overexposed is the reference viewpoint, there is a problem that the encoding efficiency is reduced as described above.

  An object of the present invention is to solve this problem and provide a technique capable of efficiently performing coding using inter-view prediction.

  In order to solve the above problems, the encoding apparatus of the present invention has the following configuration. That is, an encoding device that encodes a multi-viewpoint image captured from a plurality of viewpoints, wherein a plurality of images captured for each of the plurality of viewpoints using a shooting parameter set for each of the plurality of viewpoints. The acquisition means for acquiring the multi-viewpoint image, and when encoding the encoding target image of the plurality of images, the shooting parameter of the plurality of images is the shooting of the encoding target image. Using the image within a predetermined range based on the parameters and having the smallest parallax with the viewpoint where the encoding target image is imaged among the images within the predetermined range as the reference image, the code Encoding means for encoding the encoding target image.

  According to the present invention, the efficiency of encoding using inter-view prediction can be improved.

2A and 2B illustrate a configuration of an imaging apparatus according to a first embodiment. FIG. 3 is a diagram illustrating the arrangement of image pickup elements that constitute the optical element unit according to the first embodiment. The figure explaining the structure of the encoding part of Embodiment 1. The figure explaining the structure of the estimation part of Embodiment 1. The figure explaining the flowchart for the reference image selection of Embodiment 1. FIG. 4 is a diagram for explaining a reference relationship of captured images according to the first embodiment. FIG. 10 is a diagram for explaining a flowchart for selecting a reference image according to the second embodiment. FIG. 2 is a block diagram illustrating a configuration example of computer hardware to which the first and second embodiments can be applied.

<Embodiment 1>
In the present embodiment, an imaging apparatus having an encoding unit that encodes a multi-viewpoint image obtained by capturing a subject from a plurality of viewpoints will be described. In the present embodiment, an image read from an imaging element included in an imaging apparatus such as a digital camera or a digital camcorder, and an image received via a network are set as encoding targets. In addition, this invention is not limited to this, You may provide an encoding part in an apparatus different from an imaging device. That is, the present invention may be divided into an imaging apparatus having a plurality of imaging elements and an image encoding apparatus that performs encoding.

  FIG. 1 is a diagram illustrating an image pickup apparatus based on a multi-view system including a plurality of image pickup elements.

  In FIG. 1, the imaging apparatus 101 includes an optical element unit 102, an imaging unit 103, a CPU 104, an image processing unit 105, an encoding unit 106, a bus 107, a memory 108, a storage medium 109, and a communication unit 110.

  The optical element unit 102 controls the imaging system such as focusing on a subject, opening / closing a shutter, and adjusting an aperture value. The optical element unit 102 receives light information of a subject by a sensor (imaging element) and transfers the received signal to the imaging unit 103. FIG. 2 is a diagram illustrating a configuration example of the optical element unit 102. The optical element unit 102 includes nine image pickup elements 201 to 209 and is equally arranged on a square lattice. In FIG. 2, nine image sensors are arranged on a square lattice as an example, but the number and arrangement of image sensors are not limited to this in the present invention.

  The imaging unit 103 performs A / D conversion on the optical signal (analog data) transferred from the optical element unit 102 to acquire a plurality of images (digital data).

  The CPU 104 is a central processing unit (CPU) and comprehensively controls each unit described below.

  The image processing unit 105 performs various image processing such as white balance processing, gamma processing, and noise reduction processing on the image received from the imaging unit 103 via the bus 107.

  The encoding unit 106 encodes an image.

The bus 107 is a bus serving as a transfer path for various data. For example, digital data acquired by the imaging unit 103 is sent to a predetermined processing unit via the bus 107.
A memory 108 is a RAM and functions as a main memory, a work area, and the like of the CPU 104.

  The storage medium 109 is a removable storage medium that stores an image encoded by the encoding unit 106 (hereinafter referred to as an encoded image). Examples of the storage medium include a hard disk, a memory card, a CF card, an SD card, and a USB memory.

  The communication unit 110 transmits the encoded image stored in the storage medium 109 to a computer, a printer, a data storage server, or the like. The communication unit 110 is an interface such as USB or wireless when receiving an encoded image or a program from an external device.

  Next, a method for encoding an image captured by the imaging apparatus 101 from multiple viewpoints and transferring the image to the server will be described.

  The CPU 104 transmits photographing such as an exposure time, a focus position, and an aperture value set in advance by the user to the optical element unit 102. When the CPU 104 receives a user operation or an external shooting start instruction, the CPU 104 transmits a light reception instruction to the optical element unit 102 and transmits an image acquisition instruction to the imaging unit 103. The optical element unit 102 controls the imaging system based on the received imaging condition parameters (hereinafter referred to as imaging parameters) and a light reception instruction.

  The optical element unit 102 receives the light information of the subject at each of the imaging elements 201 to 209 at a designated frame rate, and transfers the received signal to the imaging unit 103.

  The imaging unit 103 A / D converts a plurality of received signals, and transmits the acquired image to the memory 108 via the bus 107.

  A series of imaging processes by the CPU 104, the optical element unit 102, the imaging unit 103, and the imaging elements 201 to 209 described above are continued until the imaging apparatus 101 receives an imaging end instruction from the CPU 104. For example, when the optical element unit 102 includes nine imaging elements 201 to 209 as illustrated in FIG. 2, the optical element unit 102 sets shooting parameters for each of the imaging elements 201 to 209. Each of the image sensors 201 to 209 of the optical element unit 102 receives light information from the subject, and then transmits received light signals of nine viewpoints to the image capturing unit 103. Further, each of the imaging elements 201 to 209 continues to receive light information at a designated frame rate and continues to transmit a light reception signal to the imaging unit 103. With such a multi-eye imaging device, it is possible to obtain a color image group obtained by imaging the same subject from a plurality of viewpoint positions.

  Then, the CPU 104 transmits an image processing start instruction to the image processing unit 105 after confirming that the optical element unit 102 receives light and the imaging unit 103 completes image transmission.

  When the image processing unit 105 receives an instruction to start image processing from the CPU 104, the image processing unit 105 reads an image held in the memory 108 via the bus 107. The image processing unit 105 performs various types of image processing such as development processing, white balance processing, gamma processing, and noise reduction processing. The image subjected to the image processing by the image processing unit 105 is transmitted again to the memory 108 via the bus 107 and stored.

  Further, the CPU 104 transmits an encoding process start instruction to the encoding unit 106. When receiving an encoding process start instruction from the CPU, the encoding unit 106 reads an image-processed image from the memory 108 via the bus 107 and performs an encoding process using a predetermined method. Note that the image encoding apparatus according to the present embodiment uses the encoding unit 106 to input H.264 video from a plurality of viewpoints. However, the present invention is not limited to this. That is, any encoding method based on a differential encoding method typified by MPEG or the like may be used. Details of the encoding unit 106 will be described later. Then, the encoding unit 106 transmits the encoded image (encoded image) to the storage medium 109 via the bus 107.

  The storage medium 109 stores and holds the encoded image received from the encoding unit 106.

  When the CPU 104 transmits a communication start instruction to the communication unit 110, the communication unit 110 reads the encoded image stored in the storage medium 109 via the bus 107, an external device such as a server connected to the imaging device 101, Send to storage media. Here, the communication unit 110 reads and transmits the encoded image from the storage medium 109, but the present invention is not limited to this. For example, the encoded image encoded by the encoding unit 106 may be stored and held in the memory 108, and the communication unit 110 may read the encoded image from the memory 108 and transmit it.

  A server (not shown) that is a reception destination decodes the received encoded image and reconstructs the image. The reconstructed images are taken at different viewpoints and exposure times. For this reason, the server adjusts the gradation and position of the portion of interest on each of these decoded images, and performs HDR (High Dynamic Range) composition. In HDR synthesis, a plurality of images obtained by changing exposure (or exposure) are synthesized at an arbitrary ratio, tone mapping is performed, and the dynamic range is reduced to create an HDR image with less whiteout or blackout. In the present embodiment, HDR synthesis is performed on the reconstructed image obtained by decoding the encoded image, but the present invention is not limited to the processing method of the reconstructed image.

  Note that the imaging apparatus 101 according to the present invention may include the above-described components and further other elements.

  Hereinafter, the configuration of the encoding unit 106 will be described in detail with reference to FIG.

  The encoding unit 106 includes a prediction unit 301, a transform quantization unit 302, a variable length encoding unit 303, an inverse transform inverse quantization unit 304, a deblocking filter unit 305, and a reference image memory 306.

  The prediction unit 301 is a prediction unit that predictively encodes an input image. Details of the prediction unit 301 will be described later.

  The transform quantization unit 302 is a transform quantization unit that performs orthogonal transform and quantization on the image that is predictively encoded by the prediction unit 301.

  The variable length coding unit 303 is a variable length coding unit that performs variable length coding on the transform coefficient of the image quantized by the transform quantization unit 302 and outputs a bit stream. The variable length coding unit 303 also performs variable length coding on various coding parameters including a motion vector or a disparity vector output from the prediction unit 301. Further, the variable length coding unit 303 outputs a variable length coded bit stream, and transmits the bit stream to the storage medium 109 via the bus 107.

  The inverse transform inverse quantization unit 304 is an inverse transform / inverse quantization unit that performs inverse transform by inversely quantizing the transform coefficient of the image quantized by the transform quantizing unit 302.

  The deblocking filter unit 305 is a deblocking filter that removes block distortion from the image inversely transformed by the inverse transform inverse quantization unit 304.

  The reference image memory 306 holds an image output from the deblocking filter unit 305 because it may be used as a reference image in predictive coding of subsequent images. In the present embodiment, the reference image memory 306 is provided as a dedicated memory inside the encoding unit 106, but the present invention is not limited to this. For example, a frame memory area may be allocated on the memory 108 in FIG.

  Hereinafter, the configuration of the prediction unit 301 will be described in detail with reference to FIG.

  The prediction unit 301 includes an intra encoding unit 401, an interframe encoding unit 402, and a prediction mode selection unit 403. In the present embodiment, an image is input from the memory 108 to the intra encoding unit 401 and the interframe encoding unit 402 one by one.

  The intra coding unit 401 performs intra prediction coding (hereinafter referred to as intra prediction) for performing prediction on an image output from the memory 108 using a pixel signal in the image, and outputs a prediction error.

  On the other hand, the inter-frame coding unit 402 performs inter-frame prediction coding (hereinafter, inter-frame prediction) on the image input from the memory 108 using the reference image held in the reference image memory 306. Note that inter-frame prediction includes encoding using the same viewpoint image as a reference image (hereinafter referred to as the same viewpoint prediction) and inter-view prediction using an image captured at a different viewpoint such as an adjacent viewpoint image as a reference image. is there. Therefore, the inter-frame coding unit 402 performs the same viewpoint prediction or the inter-view prediction on the input image. Then, the inter-frame coding unit 402 outputs a motion vector or a disparity vector and a prediction error.

  The prediction mode selection unit 403 selects one of the image intra-predicted by the intra-coding unit 401 or the image predicted by the inter-frame coding unit 402, and predicts the prediction error and the prediction of the selected image. Outputs the mode. Note that if it is set in advance whether intra prediction or inter-frame prediction is performed on the input image, selection is not necessary.

  The interframe coding unit 402 includes a shooting parameter acquisition unit 405, a reference image selection unit 406, and a vector search unit 407.

  The shooting parameter acquisition unit 405 acquires shooting parameters such as an exposure time, a focus position, and an aperture value in each of the image sensors 201 to 209 from the memory 108. The shooting parameter acquisition unit 405 stores and holds shooting parameters in association with the position information of the image sensors 201 to 209. The imaging parameter acquisition unit 405 outputs the imaging parameters acquired from the memory 108 to the reference image selection unit 406 when performing inter-viewpoint prediction.

  The reference image selection unit 406 uses the shooting parameters received from the shooting parameter acquisition unit 405 to select an image to be added to the reference image list. The reference image list is an image group used as a reference image when performing a vector search on an image to be encoded (hereinafter referred to as an encoding target image). Then, the reference image selection unit 406 transmits the created reference image list to the vector search unit 407.

  Based on the reference image list received from the reference image selection unit 406, the vector search unit 407 reads a reconstructed image that becomes a reference image candidate (hereinafter referred to as a reference image candidate) from the reference image memory 306. Then, the vector search unit 407 performs a vector search using the input encoding target image and the reference image candidate, and selects a reference image used for inter-frame prediction. Then, the vector search unit 407 outputs the disparity vector, the motion vector, and the prediction error to the prediction mode selection unit 403.

  In the present embodiment, the shooting parameter acquisition unit 405 stores and holds the shooting parameters of the imaging devices 201 to 209. However, in the present invention, the reference image selection unit 406 may store the shooting parameters in order to determine the reference image.

  Next, a method for selecting an image to be added to the reference image list using the imaging parameters of the imaging element that has captured the encoding target image in the reference image selection unit 406 will be described with reference to the flowchart of FIG. In the flowchart of FIG. 5, an image captured by an image sensor that has close imaging parameters to the image sensor that captured the encoding target image and has a small parallax with the image sensor is selected. Here, it is assumed that the image sensor corresponding to the reference viewpoint is the image sensor 205 and the encoding target image is captured by the image sensor 202. Note that the present embodiment is not limited to this.

  In step S501, it is determined whether or not the encoding target image is an image captured from a reference viewpoint (hereinafter referred to as a reference viewpoint image). If the encoding target image is a reference viewpoint image in step S501, the inter-viewpoint prediction is not performed, and thus the determination process ends. On the other hand, when the encoding target image is not the reference viewpoint image, the process of step S502 is performed. In the present embodiment, the reference viewpoint is the image sensor 205, and the image captured by the image sensor 202 is not a reference viewpoint image. Therefore, the process proceeds to step S502.

  In step S <b> 502, the imaging parameter acquisition unit 405 acquires the imaging parameters of the image sensor 202 that has captured the encoding target image. The shooting parameters are parameters used at the time of shooting, such as exposure time, focus position, aperture value, lens brightness (F value), and sensor sensitivity value. In the present invention, an image to be added to the reference image list is selected using the shooting parameters of each image sensor, but at least one item of the above-described shooting parameters may be used. Here, in order to simplify the description, it is assumed that the shooting parameter is the exposure time.

  In step S503, the image sensor 202 of the encoding target image and an image sensor that falls within a predetermined parallax threshold range are selected. In the present invention, the parallax in the present embodiment is a parameter based on the distance between the image sensor that has captured the encoding target image and each image sensor. For example, parallax levels such as level 1, level 2, and level 3 may be given in order of increasing distance between images. Note that the present invention is not limited to this, and the parallax may be the distance between the image pickup devices. In addition, the parallax may be an angle indicating the difference between the viewpoints of the imaging element that captured the encoding target image and the subject of each imaging element other than the imaging element. Further, in the present embodiment, the CPU 104 sets the parallax of each of the image sensors 201 to 209 set in advance by the user or the external interface in the reference image selection unit 406 in the encoding unit 106. In the present invention, the present invention is not limited to this. For example, the output source of the parallax may be the optical element 102 or the imaging unit 103.

  Further, in step S503, the ID of the extracted image sensor is acquired, and the image captured by each selected image sensor is set as a candidate image (hereinafter, list candidate) to be added to the reference image list. That is, when there are N selected image sensors, there are N list candidates. Then, these N list candidates are combined into a list candidate group. The processing described later is performed in an appropriate order for each image of the N list candidate groups, and a reference image list is finally created.

  In steps S504 to S508, an image taken with an imaging device that satisfies a predetermined condition for the exposure time and has the smallest parallax is selected as a reference image for predictive coding. The reference image is not limited to the reference image selected in this way, and can be appropriately selected from a combination of exposure time and parallax. For example, even if the parallax does not satisfy the parallax threshold, those with the same or closest exposure time may be selected as the reference image. This is effective when there are few viewpoint images with the same or closest exposure time. First, in step S504, the exposure time and the parallax of the image sensor to be processed n-th (1 ≦ n ≦ N) among all N (N ≧ 1) image sensors extracted in step S503 are acquired. n represents the number of times the determination process has been performed (hereinafter referred to as the number of determinations). Furthermore, the exposure time of the n-th image sensor is exposure time Tn, and the parallax between the n-th image sensor and the image sensor 202 to be encoded is parallax Pn.

  In step S505, it is determined whether or not the acquired exposure time Tn is within the threshold range of the exposure time. The threshold value of the exposure time is predetermined, and the lower limit threshold value of the exposure time is set as the lower limit exposure time Th1, and the upper limit threshold value of the exposure time is set as the upper limit exposure time Th2. The upper limit exposure time Th2 must be set to a value that is the same as or greater than the lower limit exposure time Th1. The lower limit exposure time Th1, the upper limit exposure threshold value, and Th2 are exposure times used for capturing an encoding target image, or times determined based on the exposure time. In step S505, it is determined whether or not the exposure time Tn is within the range between the lower limit exposure time Th1 and the upper limit exposure time Th2, which are threshold values of the two exposure times. If the exposure time Tn is within the threshold range, the process proceeds to step S506, and the parallax Pn is determined. On the other hand, when the exposure time Tn is outside the threshold range, the image captured by the nth image sensor is not added to the reference image list. Then, the process proceeds to step S508, and it is determined whether or not an image captured by the next image sensor (n + 1-th image sensor) is added to the reference image list.

  In step S506, it is determined whether or not the parallax of the image sensor corresponding to each list candidate is smaller than a threshold value. Specifically, the parallax Pn between the n-th imaging element and the imaging element 202 to be encoded is compared with the minimum parallax Pmin that is a parallax threshold, and it is determined whether or not the parallax Pn is smaller than the minimum parallax min. Note that the minimum parallax Pmin when the number of determinations is 1 (n = 1) is a parallax initial value Ph that is a threshold value determined in advance as an initial value. If the parallax Pn is smaller than the minimum parallax Pmin, the process proceeds to step S507. On the other hand, when the parallax Pn is equal to or greater than the minimum parallax Pmin, the image captured by the nth image sensor is not added to the reference image list. Then, the process proceeds to step S508, and it is determined whether or not an image captured by the next image sensor is added to the reference image list. In step S507, the image captured by the n-th image sensor is held as a reference image list, and the parallax Pn between the n-th image sensor and the image sensor 202 to be encoded is rewritten as the minimum parallax Pmin that is a parallax threshold. .

  In step S <b> 508, the reference image selection unit 406 determines whether to finish selecting the image to be added to the reference image list. Specifically, when the number of determinations n is smaller than N, the process returns to step S504 to repeat the determination process via step S509 in order to create an optimal reference image list. On the other hand, when the number of determinations n is equivalent to N, it is determined that the reference image selection processing in encoding of the encoding target image has been completed for all list candidate groups, and the determination processing ends.

  If a list candidate already exists in the reference image list in step S507, the list candidate previously stored and retained is discarded, and a list candidate having a smaller parallax is retained in the reference image list.

  In the present embodiment, the flowchart of FIG. 5 shows a procedure for adding an image captured by an image sensor that captures an encoding target image close to the image capturing parameter and has a small parallax with the image sensor to the reference image list. However, the present invention is not limited to this. For example, when a plurality of reference images are selected and predictive coding is performed using these, in addition to the images selected according to the flowchart of FIG. May be added.

  In this embodiment, the exposure time is used as the shooting parameter, but the present invention is not limited to this. For example, using the exposure time T, the aperture value I, the lens brightness F, and the sensor sensitivity S, a value C that represents a shooting parameter may be calculated as in Expression (1).

C = a1 * T + a2 * I + a3 * F + a4 * S- (1)
However, a1, a2, a3, and a4 of Formula (1) are predetermined constants.

  With the above configuration and operation, the reference image selection unit 406 narrows down the reference image candidates when inter-viewpoint prediction is performed on images captured from a plurality of viewpoints with different exposure times for each image sensor. Can do. Therefore, it is possible to reduce the calculation amount of the disparity vector.

  In this embodiment, in step S503 of the process in the reference image selection unit 406, the image sensor 202 of the image to be encoded and an image sensor that falls within a predetermined parallax threshold range are selected and set as a list candidate group. However, in the present invention, step S503 may be omitted. That is, a process for creating a reference image list for all images other than the encoding target image at the same time may be performed without selecting a list candidate group. The processing in step S503 is provided in order to omit processing of an image shot by an imaging device far from the imaging device 202 of the encoding target image. Accordingly, it is not necessary to perform the processing in steps S505 to S509 for an image that is outside the range of the parallax threshold. That is, the amount of calculation can be reduced as compared with the case where all the images other than the encoding target image are processed.

  In the present embodiment, a list candidate group is selected from all the image sensors and a reference image list is created. However, the present invention is not limited to this. That is, the list candidate group and the reference image list are not necessarily required, and those that are not used as reference images may be removed from the ID lists of all the image pickup devices.

  Hereinafter, a method for selecting an image to be added to the reference image list will be specifically described.

  Here, as an example, the number of image sensors in the optical element unit 102 is six. Further, the lower limit exposure time Th1 and the upper limit exposure time Th2 of the exposure time threshold are both equal to the exposure time of the imaging device that has captured the encoding target image, and the initial parallax value Ph that is the initial parallax threshold is 5 (initial The minimum parallax Pmin = the initial parallax value Ph = 5 Note that the same value is set for the lower limit exposure time Th1 and the upper limit exposure time Th2 in order to simplify the explanation, and the present invention is limited to this. It is not a thing.

  Also, # 1 to # 6 are assigned as IDs of six image sensors. Table 1 shows exposure times set for the image sensors # 1 to # 6.

図６は、各撮像素子＃１〜＃６を用いて撮影された画像を撮影時刻毎に並べ、フレーム間符号化部４０２によるフレーム間予測をする際の画像の参照関係を示した図である。本実施形態では、図６において、撮像素子♯１で撮影された画像を基準視点画像とする。また、符号化部１０６は、各撮像素子＃１〜＃６から出力される画像を、撮像素子＃１、＃２、＃３、＃４、＃５、＃６に対応する画像の順番で符号化する。尚、本発明はこれに限定されるものではない。例えば、撮像素子＃３から出力される画像を基準視点画像としてもよいし、各撮像素子＃１〜＃６に対応する画像を上記と異なる順番で符号化してもよい。

FIG. 6 is a diagram illustrating an image reference relationship when images captured using the image sensors # 1 to # 6 are arranged at each imaging time and interframe encoding is performed by the interframe encoding unit 402. . In the present embodiment, in FIG. 6, an image captured by the image sensor # 1 is set as a reference viewpoint image. The encoding unit 106 encodes the images output from the image sensors # 1 to # 6 in the order of the images corresponding to the image sensors # 1, # 2, # 3, # 4, # 5, and # 6. Turn into. Note that the present invention is not limited to this. For example, an image output from the image sensor # 3 may be used as a reference viewpoint image, and images corresponding to the image sensors # 1 to # 6 may be encoded in an order different from the above.

  First, taking as an example the image B41 imaged by the image sensor # 4 at time t1 in FIG. 6, the reference image selection unit 406 in FIG. 4 shows a processing method for creating an image list in FIGS. Will be described.

  In step S501, the reference image selection unit 406 determines whether the image B41 that is the encoding target image is a reference viewpoint image. Since the image B41 is not the reference viewpoint image, the process proceeds to step S502.

  In step S <b> 502, the reference image selection unit 406 acquires the imaging parameter of the image sensor # 4 that has captured B <b> 41 from the imaging parameter acquisition unit 405. As shown in Table 1, the exposure time of the image sensor # 4 is 1/125.

  In step S <b> 503, the reference image selection unit 406 sets an image captured by an imaging device having a parallax initial value Ph smaller than 5 as a list candidate. List candidates for encoding the image B41 are selected from previously encoded images. Further, at time t1, an image photographed at time t1 at the same time with an image sensor close to the parallax of image sensor # 4 among image sensors # 1 to # 6 can be a list candidate. That is, the list candidate group in the encoding of the image B41 includes the image I11, the image P21, and the image P31. For this reason, the IDs of the image sensors corresponding to the list complement group are # 1, # 2, and # 3. That is, the number N of image sensors corresponding to the images listed in the list candidate group is 3 (N = 3). Here, in order to simplify the description, in steps S504 to S507, the process of determining the IDs of the image sensors is performed in ascending order (the order of # 1, # 2, and # 3).

  In step S504, the exposure time and the parallax of the first image sensor # 1 are acquired. From Table 1 and FIG. 6, the exposure time of the image sensor # 1 is 1/125, and the parallax is 3.

  In step S505, it is determined whether or not the exposure time 1/125 of the image sensor # 1 is within the threshold value of the exposure time. Both the lower limit exposure time Th1 and the upper limit exposure time Th2 of the threshold of the exposure time are values equal to the exposure time 1/125 of the image sensor # 4 that captured the encoding target image. Therefore, since the exposure time 1/125 of the image sensor # 1 is within the threshold range, the process proceeds to step S506.

  In step S506, it is determined whether or not the parallax of the image sensor # 1 is smaller than the minimum parallax Pmin. Since the minimum parallax Pmin at this time is equal to the initial parallax value Ph, the minimum parallax Pmin is 5. Therefore, the process proceeds to step S507.

  In step S507, the image I11 is added to the reference image list, and the minimum parallax Pmin, which is the parallax threshold, is changed from the parallax initial value Ph = 5 to the parallax value 3 of the optical element # 1 (rewritten to Pmin = 3). Then, the process proceeds to step S508.

  In step S508, it is determined whether reference image selection processing in encoding of the image to be encoded has been completed for all list candidate groups. Since the number of determinations is 1 (n = 1) at this time, n <N (N = 3). Therefore, it is determined that processing has not been completed for all list candidate groups. Then, the process returns to step S504 via step S509, and the determination process is repeated for the image P21 photographed by the image sensor # 2 that is the next list candidate. However, in step S504, since the exposure time of the image sensor # 2 is not within the threshold range, the image P21 is not added to the reference image list. For the same reason, the image P31 captured by the image sensor # 3 is not added to the reference image list.

  That is, the image I11 is held in the reference image list in the inter-frame prediction of the image B41 captured by the image sensor # 4 at time t1 in FIG. Then, the vector search unit 407 performs a vector search using the image I11.

  Note that the present invention is not limited to this, and in the case where a plurality of images are held in the reference image list, in addition to the image I11, for example, the image P31 that is the image with the smallest parallax is also added to the reference image list. Also good.

  Next, as another example, a processing method for creating a reference image list of an image B64 photographed by the image sensor # 6 at time t4 in FIG. 6 will be described with reference to FIGS. .

  In step S502, the exposure time 1/2000 of the image sensor # 6 that captured the image B64 is acquired.

  In step S503, an image sensor listed in the list candidate group is selected. The images selected as the list candidate group are four images B24, B34, B44, and B54 that satisfy the parallax threshold among the images shot at the same time as the image B64 (the number of imaging elements N = 4). .

  Then, the determination processes in steps S504 to S507 are performed on each of the image sensors # 2, # 3, # 4, and # 5 corresponding to each list candidate group.

  As a result, the reference image list holds the image B34 taken at # 3. This is because the image B34 satisfies the exposure time threshold value in step S505.

  Note that the present invention is not limited to this. That is, the image B54 having the smallest parallax among the images taken at the same time, that is, the time t4 when B64 was taken may be added to the reference image list. In addition, the image B63 and the image B65 captured at different times by the imaging element # 6 that captured B64 may be added to the reference image list.

<Embodiment 2>
In the first embodiment, the method of selecting the reference image using the imaging parameter of the imaging element that has captured the encoding target image in the reference image selection unit 406 has been described with reference to the flowchart of FIG. In the present embodiment, another method for selecting a reference image in the reference image selection unit 406 will be described using the flowchart shown in FIG.

  The imaging apparatus according to the present embodiment has the same configuration as that of the first embodiment illustrated in FIGS. 1, 2, 3, and 4, and thus description thereof is omitted.

  Hereinafter, a method of selecting a reference image using the imaging parameters of the image sensor that has captured the encoding target image will be described with reference to the flowchart of FIG. However, step S701, step S702, step S703, step S706, and step S708 in FIG. 7 correspond to step S501, step S502, step S503, step S505, and step S509 in FIG. 5, respectively. Therefore, detailed description is omitted.

  Steps S701 to S703 execute the same operations as steps S501 to S503 in FIG.

  In step S704, parallaxes of all N (N ≧ 1) image sensors selected in step S703 are acquired. In step S704, the processing order is set for the N imaging elements in the order of the acquired parallax.

  In step S705, based on the processing order set in step S704, the exposure time and the parallax of the image sensor to be processed nth (1 ≦ n ≦ N) are acquired. Also, let the exposure time of the n-th image sensor be the exposure time Tn, and the parallax based on the distance from the image sensor that captured the encoding target image be the parallax Pn.

  In step S706, it is determined whether or not the exposure time Tn acquired in step S705 is within the exposure time threshold range. The threshold values for the exposure time are the lower limit exposure time Th1 and the upper limit exposure time Th2. In step S706, it is determined whether or not the exposure time Tn is within the range between the lower limit exposure time Th1 and the upper limit exposure time Th2, which are the threshold values of the two exposure times. If the exposure time Tn is within the threshold range, the process proceeds to step S707. On the other hand, when the exposure time Tn is outside the threshold range, the image captured by the nth image sensor is not added to the reference image list. Then, the process proceeds to step S708, and it is determined whether or not an image captured by the next image sensor (n + 1-th image sensor) is added to the reference image list.

  In step S707, the image captured by the n-th image sensor is held as a reference image list, and the determination of the reference image selection process in the reference image selection unit 406 is ended.

  In the present embodiment, in step S704, the processing order is set in ascending order of parallax, and the imaging parameters are compared based on the processing order. Thus, it is not necessary to determine whether or not the shooting parameters satisfy the threshold for all the list candidate groups. Further, in the present embodiment, a procedure for adding an image captured by an image sensor that has close imaging parameters to the image sensor that captured the encoding target image and that has a small parallax with the image sensor to the reference image list is illustrated in FIG. This has been described with reference to the flowchart of FIG. However, the present invention is not limited to this. For example, when a plurality of reference images are selected and predictive coding is performed using these, in addition to the images selected according to the flowchart of FIG. May be added.

<Embodiment 3>
In the first and second embodiments, the imaging apparatus that executes the present invention has been described. However, the present invention is not limited to the imaging apparatus, and can be executed in an image processing apparatus having an image processing unit and an encoding unit. Even if the image processing unit and the encoding unit are not provided, the processing performed by these processing units may be executed by a computer program.

  FIG. 8 is a block diagram illustrating a configuration example of computer hardware to which the above-described embodiments can be applied.

  The CPU 801 controls the entire computer using computer programs and data stored in the RAM 802 and the ROM 803, and executes each process described above. That is, the CPU 801 functions as the image processing unit and the encoding unit illustrated in FIG.

  The RAM 802 has an area for temporarily storing computer programs and data loaded from the external storage device 806, data acquired from the outside via an I / F (interface) 807, and the like. Further, the RAM 802 has a work area used when the CPU 801 executes various processes. That is, the RAM 802 can be allocated as a frame memory, for example, or can provide other various areas as appropriate.

  The ROM 803 stores setting data of the computer, a boot program, and the like. The operation unit 804 is configured by a keyboard, a mouse, and the like, and can input various instructions to the CPU 801 by the user of the computer.

  The external storage device 806 is a mass information storage device represented by a hard disk drive device. The external storage device 806 stores an OS (Operating System) and computer programs for causing the CPU 801 to realize the functions of each unit. Furthermore, each image data as a processing target may be stored in the external storage device 806.

  Computer programs and data stored in the external storage device 806 are appropriately loaded into the RAM 802 under the control of the CPU 801 and are processed by the CPU 801. The I / F 807 can be connected to a network such as a LAN or the Internet, and other devices such as a projection device and a display device. The computer can acquire and send various information via the I / F 807. Can be. A bus 808 connects the above-described units.

  The operation having the above-described configuration is controlled by the CPU 801 centering on the operation described in the above flowchart.

<Other embodiments>
The object of the present invention can also be achieved by supplying a storage medium storing a computer program code for realizing the above-described functions to the system, and the system reading and executing the computer program code. In this case, the computer program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the computer program code constitutes the present invention. In addition, the operating system (OS) running on the computer performs part or all of the actual processing based on the code instruction of the program, and the above-described functions are realized by the processing. .

  Furthermore, you may implement | achieve with the following forms. That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Then, based on the instruction of the code of the computer program, the above-described functions are realized by the CPU or the like provided in the function expansion card or function expansion unit performing part or all of the actual processing.

  When the present invention is applied to the above storage medium, the computer program code corresponding to the flowchart described above is stored in the storage medium.

Claims (8)

An encoding device that encodes a multi-viewpoint image captured from a plurality of viewpoints,
An acquisition unit configured to acquire the multi-viewpoint image including a plurality of images captured for each of the plurality of viewpoints using imaging parameters set for the plurality of viewpoints;
When encoding the encoding target image of the plurality of images, the shooting parameter of the plurality of images is an image within a predetermined range based on the shooting parameter in the encoding target image, And encoding means for encoding the encoding target image using, as a reference image, an image having the smallest parallax with respect to a viewpoint where the encoding target image is imaged, among the images within the predetermined range. An encoding apparatus comprising: The encoding apparatus according to claim 1, wherein the parallax is based on a distance between the plurality of viewpoints. The encoding apparatus according to claim 1, wherein the imaging parameter is at least one of an exposure time, a focus position, an aperture value, a lens brightness, and a sensor sensitivity value. The imaging device that captures the multi-viewpoint image includes a plurality of imaging elements,
The encoding according to any one of claims 1 to 3, wherein the multi-viewpoint image is an image captured from the plurality of viewpoints with respect to the subject using the plurality of imaging elements. apparatus. The encoding apparatus according to any one of claims 1 to 4, wherein the encoding unit performs inter-frame predictive encoding on the encoding target image using the reference image. The encoding device according to any one of claims 1 to 5,
An imaging apparatus comprising: a plurality of imaging units that capture the multi-viewpoint images. An encoding method for encoding a multi-viewpoint image captured from a plurality of viewpoints,
An acquisition step of acquiring the multi-viewpoint image including a plurality of images captured for each of the plurality of viewpoints using imaging parameters set for the plurality of viewpoints;
When encoding the encoding target image of the plurality of images, the shooting parameter of the plurality of images is an image within a predetermined range based on the shooting parameter in the encoding target image, And an encoding step of encoding the encoding target image using, as a reference image, an image having the smallest parallax with respect to a viewpoint where the encoding target image is captured among images within the predetermined range. An encoding method comprising:   A program for causing a computer to function as each unit of the encoding device according to any one of claims 1 to 5.

Images