JP6110724B2 - Image processing apparatus, encoding apparatus, and encoding program - Google Patents

Description

  The present invention relates to an image processing apparatus, an encoding apparatus, and an encoding program that process an image based on image depth information.

    In recent years, a camera equipped with a depth sensor capable of acquiring depth information has been actively researched and developed. For example, there is Microsoft's Kinect (registered trademark) as an example of a camera equipped with an RGB-D sensor for acquiring image information obtained by adding D (depth) to RGB. From such a background, it is expected that the use of moving image information having a two-dimensional image signal and depth information will increase.

  In addition, H.264, which is a standard for efficiently processing moving image information. H.264 / AVC (Advanced Video Coding) and H.264. In the H.265 / HEVC (High Efficiency Video Coding) video compression encoding method, each frame of the video is divided into rectangular areas called blocks, and encoding is performed.

  As one of the video compression methods, image prediction using a block motion vector is used. The motion vector is a value indicating a spatial deviation in units of blocks obtained using a block matching technique or the like that searches for a position most similar to the processing target block from the reference image in units of blocks. By obtaining a difference value between the predicted image and the image to be processed using this motion vector, it is possible to reduce the amount of video data.

  In the above standards, motion vectors are also used. Note that the motion vector detection method itself is not defined as a standard, but is performed using, for example, a block matching method for each block (see, for example, Non-Patent Document 1).

  FIG. 1 shows an example of motion vector detection. The moving image information includes a plurality of frames. For example, the reference frame 110 represents a moving image frame at time t = 0. The reference frame 120 represents a moving image frame at time t = −1. Each frame is composed of a plurality of blocks. In order to search for a motion vector of the processing target block 112 existing in the reference frame 110, a search range 121 is set in the reference frame 120. When the block matching method is used, the image of the block to be processed on the base frame and the image SAD (Sum of Absolute Distance: prediction error) of the block within the search range set around the same spatial position on the reference frame (Sum of Square Distance) or SSD (Sum of Square Distance). A motion vector is determined from the position difference between the block on the reference frame having the smallest value and the processing target block on the base frame.

  H. H.264 / AVC and H.264. In the H.265 / HEVC standard, the base frame and the reference frame are vertically or horizontally doubled or quadrupled, and motion vector detection with 1/2 or 1/4 pixel accuracy is performed as viewed from the original image.

  Since the motion vector detection accuracy affects the compression efficiency of the video, a motion vector detection method with higher accuracy is desired.

Supervised by Satoshi Okubo, “Revised 3rd edition H.264 / AVC textbook”, Impress R & D, p124, January 1, 2009

  As described above, in motion vector detection using a technique such as block matching, although the processing target block of the base frame and the block of the reference frame belong to different objects, the luminance or color of both approximates each other. Therefore, the result of block matching may be minimized. In this case, the detected motion vector does not reflect the original motion of the object. In particular, H.C. H.264 / AVC and H.264. When the temporal direct mode is adopted in the standard of H.265 / HEVC, a motion vector between two adjacent frames may be predicted, for example, by scaling the motion vector of the adjacent frame by two times. . In this case, since the objects are different from each other, the motion vector between the two adjacent frames is further deviated from the original motion vector. The result appears as a reduction in the encoded image quality.

  Although it is possible to determine whether or not they are the same object using, for example, color information, the determination accuracy has a limit. For example, even if the objects are different, if these objects have the same color, these objects are determined to be the same object.

  Therefore, the present invention provides an image processing device, an encoding device, and an encoding program that can improve image quality by detecting motion vectors with higher accuracy using depth information. Objective.

  An image processing apparatus according to an aspect of the present invention is an image processing apparatus that performs processing in units of blocks on an image including depth information, and includes a processing target block that is one of a plurality of blocks of a reference frame; Based on the matching degree of the image of the reference frame to be referenced, a motion vector candidate specifying unit that specifies a plurality of motion vector candidates of the processing target block, depth information of the processing target block, and the motion vector are obtained. Using the depth information of the corresponding block of the reference frame, an object determination unit that determines whether the processing target block and the corresponding block of the reference frame belong to the same object, and the motion vector Based on the degree of coincidence corresponding to each of the plurality of candidates and the determination result of the object determination unit. Determining the motion vector of the block has a motion vector determination unit.

  In addition, a depth information clustering unit that performs clustering processing on the depth information of the base frame and the reference frame may be further included, and the object determination unit may perform the determination using a result of the clustering.

  The motion vector determination unit may belong to the same object when a block of a reference frame related to the plurality of motion vector candidates is determined to belong to the same object as the processing target block. Among the motion vectors determined as the motion vector of the processing target block, or any block of the reference frame related to a plurality of motion vector candidates When it is determined that the block does not belong to the same object as the processing target block, the motion vector having the highest degree of matching may be determined as the motion vector of the processing target block.

  The motion vector determination unit may determine that any block of the reference frame related to a plurality of motion vector candidates does not belong to the same object as the processing target block, and the processing target block is When the image processing apparatus can be further divided into a plurality of sub-blocks, the processing of the image processing apparatus may be repeated with the sub-block as a processing target block.

  An encoding apparatus according to another aspect of the present invention may include the image processing apparatus.

  An encoding program according to another aspect of the present invention causes a computer to function as the encoding apparatus.

  According to the present invention, it is possible to more accurately detect a motion vector using depth information and improve image quality.

It is a figure which shows the example of a detection of a motion vector. 1 is a block diagram illustrating an example of a configuration of an image processing apparatus 10 in Embodiment 1. FIG. It is a figure which shows the example of a detailed structure of the motion vector determination part 170. FIG. 3 is a flowchart illustrating an example of processing performed by the image processing apparatus according to the first exemplary embodiment. It is a block diagram which shows an example of schematic structure of the encoding apparatus in Example 2. FIG. 10 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus according to Embodiment 3. FIG.

  Hereinafter, each embodiment will be described in detail with reference to the accompanying drawings.

[Example 1]
First, the image processing apparatus according to the first embodiment will be described. The image processing apparatus according to the first embodiment is an apparatus that determines a motion vector using depth information. This image processing apparatus may be implemented by an integrated circuit. Further, the program may be implemented by reading a program stored in a storage medium by a computer and executing the program by the computer.

<Configuration>
FIG. 2 is a block diagram illustrating an example of the configuration of the image processing apparatus 10 according to the first embodiment. The image processing apparatus 10 illustrated in FIG. 2 includes a motion vector candidate specifying unit 140, a depth information clustering unit 150, an object determining unit 160, and a motion vector determining unit 170.

  The motion vector candidate specifying unit 140 receives image information of the base frame and the reference frame. There may be a plurality of reference frames. The motion vector candidate specifying unit 140 performs block division (for example, division into 16 × 16 pixel size) on the image of the reference frame. One of the divided blocks of the reference frame is sequentially selected as a processing target block. Then, the motion vector is detected by searching the reference frame image for the processing target block. For example, a block matching method, which is an existing method, may be used for detecting a motion vector.

  In this embodiment, it is desirable to detect a motion vector having a SAD or SSD value as small as the nth for each processing target block as a plurality of motion vector candidates. For example, 3 may be used as the value of n. The plurality of motion vector candidates are output to the motion vector determination unit 170 and the object determination unit 160. In addition, the processing target block when the motion vector is obtained and the position information of the block of the reference frame are output to the object determination unit 160.

  For example, the depth information clustering unit 150 clusters the depth information of the base frame and the reference frame. Here, the depth information exists for each pixel or for each block area determined at the time of imaging. When the depth depth indicating the depth information is, for example, 256 levels, the depth information clustering unit 150 sets the number of clusters k = 8 and applies the K-means method to cluster the depth information.

  The number of stages of depth and depth is not limited to 256 in order to follow the performance of the depth sensor. An appropriate value may be set in advance for the number k of clusters.

  In addition, the depth information clustering unit 150 may obtain a histogram of depth information, divide the range of values taken by the histogram equally into the number of clusters k, and classify the depth information.

  The depth information clustering unit 150 can use depth information clustering based on the above-described K-means method, or classification of depth information obtained from a histogram of depth information. The depth information clustering unit 150 assigns numbers in descending order of depth as the clustering result of the depth information of each pixel position or each predetermined region, and provides clustering information (for example, 0 to 7) indicating the clustering result to the object determination unit 160. Output.

  The object determination unit 160 determines whether or not the processing target block for which the motion vector is calculated and the corresponding reference frame block belong to the same object. The object determination unit 160 includes an acquisition unit 162 and a depth determination unit 164.

  The acquisition unit 162 acquires a representative value of clustering information for each block. The acquisition unit 162 acquires one depth information (representative value) for each block by averaging the clustering information for each block and performing integerization by rounding off the fractions. Note that the acquisition unit 162 may use the median value or mode value of the clustering information as a representative value for each block. The acquisition unit 162 outputs the representative value of the depth information in the block to the depth determination unit 164 as the representative depth information of the block.

  The depth determination unit 164 determines whether the depth representative information of the processing target block and the corresponding reference frame block acquired from the acquisition unit 162 is within a predetermined range. For example, the depth determination unit 164 may determine whether or not the difference between the processing target block depth information and the depth representative information of the corresponding reference frame block is within a predetermined value.

  The depth determination unit 164 can determine whether the depth information belongs to the same cluster if the predetermined value is 0, and can determine whether the depth information is similar if the predetermined value is 1, for example. .

  The depth determination unit 164 determines that the processing target block and the corresponding reference frame block belong to the same object if the difference between the depth representative information of the processing target block and the corresponding reference frame block is within a predetermined value. To do. The depth determination unit 164 also determines that the processing target block and the corresponding reference frame block belong to the same object if the difference between the processing target block and the depth representative information of the corresponding reference frame block is larger than a predetermined value. Judge that not.

  The object determination unit 160 determines the representative value directly from the depth information in the block without using the clustering information of the depth information, and represents the representative value of the depth representative information of the processing target block and the block of the corresponding reference frame. It may be determined whether the difference is within a predetermined value.

  The object determination unit 160 outputs the determination result of the depth determination unit 164 to the motion vector determination unit 170.

  The motion vector determination unit 170 uses the information from the object determination unit 160 to determine the motion vector of the processing target block from a plurality of motion vector candidates for the processing target block.

  FIG. 3 shows an example of a detailed configuration of the motion vector determination unit 170. The motion vector determination unit 170 includes a first determination unit 171, a second determination unit 172, a third determination unit 173, a selection unit 174, and a measurement unit 175.

  Note that the selection unit 174 selectively transmits the information from the motion vector candidate specifying unit 140 and the information from the object determination unit 160 to any one of the first determination unit 171, the second determination unit 172, and the third determination unit 173. May be operated to communicate to.

  In addition, the function of selectively transmitting the selection unit 174 is controlled based on information from the measurement unit described later. The selection unit 174 may be set so as to be selectively transmitted by an external setting such as an operator.

  The first determining unit 171 uses the motion vector having the highest degree of coincidence, that is, the motion vector having the smallest SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference) between the blocks as the motion vector of the processing target block. Decide whether or not to adopt. That is, when the object determination unit determines that the processing target block and the corresponding reference frame block do not belong to the same object with respect to the motion vector having the highest matching degree, the first determining unit 171 determines the matching degree. The decision may be made to reject the adoption of the motion vector having the highest value as the motion vector of the processing target block. Even if the motion vector has the highest degree of coincidence, if it is determined that the processing target block and the corresponding reference frame block do not belong to the same object, it is appropriate to adopt this motion vector. This is not a judgment.

  The first determination unit 171 outputs the motion vector having the highest degree of matching when not rejecting. When the first determination unit 171 performs rejection, null may be output. For example, in registration super-resolution processing, null is allowed as a motion vector. H. H.264 / AVC or H.264 In the standard of H.265 / HEVC, null is not appropriate, and therefore processing of the second determination unit 172 described later may be performed.

  The second determining unit 172 determines that a block of a reference frame related to a plurality of motion vector candidates belongs to the same object when there is a motion vector determined to belong to the same object as the processing target block. Among the motion vectors thus determined, the motion vector having the highest degree of coincidence is determined as the motion vector of the processing target block. In addition, when it is determined that any block of the reference frame related to the plurality of motion vector candidates does not belong to the same object as the processing target block, the second determination unit 172 selects the motion vector having the highest degree of matching, The motion vector of the processing target block may be determined.

  The third determination unit 173 is the same as the second determination unit 172 when a block of a reference frame related to a plurality of motion vector candidates is determined to belong to the same object as the processing target block. Processing may be performed. The third determining unit 173 determines that none of the blocks of the reference frame related to the plurality of motion vector candidates belong to the same object as the processing target block, and the processing target block further includes a plurality of sub-frames. If the block can be divided into blocks, the motion vector candidate specifying unit 140 may be instructed to repeat the motion vector detection processing (the processing in FIGS. 2 and 3) using the sub-block as a processing target block (176) (176). ). For example, when a vector detection process is performed using a block of 16 × 16 pixel size, this block is divided into four blocks of 8 × 8 pixel size, and similarly a motion vector detection process is performed. I do. If no further block division can be performed, the motion vector having the highest degree of matching may be determined as the motion vector of the processing target block.

  In addition, when the above further block division is performed, this instruction 176 is output to the entropy encoding unit 204 described later with reference to FIG. 5 in order to include the instruction 176 relating to this block division in the bitstream. In addition to encoding, the motion vector candidate specifying unit 140 may be instructed as described above.

  In block decoding, the encoded instruction 176 related to the block division is decoded and used.

  Further, information after the image is quantized (for example, quantized data 180 from the quantization unit 203 in FIG. 5) is input to the measurement unit 175. When the third determination unit is selected by the selection unit 174, the measurement unit 175 measures the information amount per unit time related to the quantized data 180 as needed. For example, if the information amount per unit time related to the quantized data 180 tends to increase, the selection unit may issue a command to the selection unit so that the first determination unit 171 or the second determination unit 172 is selected. Good. The reason for performing this processing is that, for example, when block division frequently occurs, the data representing the instruction 176 related to block division can increase the amount of information per unit time related to the quantized data 180. This is because it is desirable to stop the amount of information per unit time below a predetermined amount.

  With the above configuration, it is possible to detect motion vectors more accurately using motion information and improve image quality.

<Operation>
Next, the operation of the image processing apparatus 10 according to the first embodiment will be described with reference to FIGS. 4 and 5.

  FIG. 4 shows an overview of processing for one processing target block of the reference frame of the first embodiment.

  In step S102, the motion vector candidate specifying unit 140 specifies a plurality of motion vector candidates for the processing target block.

  In step S104, the depth information clustering unit 150 performs a depth information clustering process of the base frame and the reference frame. The clustering process may use the K-means method. The depth information clustering unit 150 assigns depths in descending order as the depth information clustering results, and outputs clustering information (for example, 0 to 7) indicating the clustering results for each pixel position or each predetermined region. May be.

  In step S <b> 106, the object determination unit 160 acquires, in the acquisition unit 162, a representative value representing the block depth for each related block. The acquisition unit 162 acquires one depth information (representative value) for each block by averaging the clustering information for each block and performing integerization by rounding off the fractions. Note that the acquisition unit 162 may use the median value or mode value of the clustering information as a representative value for each block.

  In step S108, the depth determination unit determines whether or not the processing target block associated with each of the plurality of motion vector candidates and the corresponding block in the reference frame belong to the same object.

  In step S110, when it is determined that the two blocks from which the motion vector of the first candidate having the highest degree of coincidence is acquired belong to the same object, the motion vector determination unit 170 determines the first candidate. The motion vector is determined as the motion vector of the processing target block.

  In cases other than the above, as described above, the operation is as follows.

  First, the first determination unit 171 outputs NULL.

  The second determining unit 172 determines that a block of a reference frame related to a plurality of motion vector candidates belongs to the same object when there is a motion vector determined to belong to the same object as the processing target block. Of the motion vectors thus determined, the motion vector having the highest degree of coincidence may be determined as the motion vector of the processing target block. In addition, when it is determined that any block of the reference frame related to the plurality of motion vector candidates does not belong to the same object as the processing target block, the second determination unit 172 selects the motion vector having the highest degree of matching, The motion vector of the processing target block may be determined.

  The third determination unit 173 is the same as the second determination unit 172 when a block of a reference frame related to a plurality of motion vector candidates is determined to belong to the same object as the processing target block. Process. The third determining unit 173 determines that none of the blocks of the reference frame related to the plurality of motion vector candidates belong to the same object as the processing target block, and the processing target block further includes a plurality of sub-frames. If it can be divided into blocks, the block is divided into sub-blocks, and this sub-block is set as a processing target block, and an instruction 176 for the process of repeating the motion vector detection process (the processes in FIGS. 2 and 3) is output.

  As described above, according to the first embodiment, by using the depth information, a motion vector can be detected based on blocks determined to be the same object, and the motion vector can be detected more accurately, thereby improving the image quality. Can be achieved.

[Example 2]
In the second embodiment, an encoding device including the image processing device 10 in the first embodiment in the motion vector calculation unit 213 will be described. In the second embodiment, it is assumed that depth information is included in input information.

<Configuration>
FIG. 5 is a block diagram illustrating an example of a schematic configuration of the encoding device 20 according to the second embodiment. In the example illustrated in FIG. 5, the encoding device 20 includes a preprocessing unit 200, a prediction error signal generation unit 201, an orthogonal transformation unit 202, a quantization unit 203, an entropy encoding unit 204, and an inverse quantization unit. 205, an inverse orthogonal transform unit 206, a decoded image generation unit 207, a loop filter unit 209, a decoded image storage unit 210, an intra prediction unit 211, an inter prediction unit 212, a motion vector calculation unit 213, and a prediction And an image selection unit 215. An outline of each part will be described below.

  The preprocessing unit 200 rearranges the pictures according to the picture type, and sequentially outputs the picture type and the frame image for each frame. The preprocessing unit 200 may also perform block division and the like, and output block division boundary information to the loop filter unit 209. In addition, the preprocessing unit 200 outputs depth information included in the original image to the motion vector calculation unit 213.

  The prediction error signal generation unit 201 obtains block data obtained by dividing the encoding target image of the input moving image data into blocks such as 32 × 32, 16 × 16, and 8 × 8 pixels, for example.

  The prediction error signal generation unit 201 generates a prediction error signal based on the block data and the block data of the prediction image output from the prediction image selection unit 215. The prediction error signal generation unit 201 outputs the generated prediction error signal to the orthogonal transformation unit 202.

  The orthogonal transform unit 202 performs an orthogonal transform process on the input prediction error signal. The orthogonal transform unit 202 outputs a signal indicating the transformed coefficient value to the quantization unit 203.

  The quantization unit 203 quantizes the output signal from the orthogonal transform unit 202. The quantization unit 203 reduces the code amount of the output signal by quantization, and outputs this output signal to the entropy encoding unit 204 and the inverse quantization unit 205. The quantization unit 203 outputs the QP value of the quantization parameter to the loop filter unit 209. The output signal of the quantization unit may be output to the motion vector calculation unit 213 as quantized data 180.

  The entropy encoding unit 204 entropy-encodes and outputs the output signal from the quantization unit 203, the motion vector information output from the motion vector calculation unit 213, the filter coefficient from the loop filter unit 209, and the like.

  Also, the entropy encoding unit 204 entropy encodes the difference value of the intra prediction direction acquired from the intra prediction unit 211, the difference value of the motion vector and the prediction vector acquired from the inter prediction unit 212, and the like.

  In addition, the entropy encoding unit 204 may encode information regarding block subdivision acquired from the third determination unit 173 in the motion vector determination unit 170. Entropy coding is a method of assigning variable-length codes according to the appearance frequency of symbols.

  The inverse quantization unit 205 performs inverse quantization on the output signal from the quantization unit 203 and outputs the result to the inverse orthogonal transform unit 206. The inverse orthogonal transform unit 206 performs an inverse orthogonal transform process on the output signal from the inverse quantization unit 205 and then outputs the output signal to the decoded image generation unit 207. By performing decoding processing by the inverse quantization unit 205 and the inverse orthogonal transform unit 206, a signal having the same level as the prediction error signal before encoding is obtained.

  The decoded image generation unit 207 is decoded by the block data of the image predicted in the screen by the intra prediction unit 211 or the motion compensated image by the inter prediction unit 212, and the inverse quantization unit 205 and the inverse orthogonal transform unit 206. Add the prediction error signal. The decoded image generation unit 207 outputs the decoded image block data generated by addition to the loop filter unit 209.

  The loop filter unit 209 is, for example, an ALF (Adaptive Loop Filter) or a deblocking filter. The loop filter unit 209 outputs the filter processing result to the decoded image storage unit 210, and stores the accumulated filter processing result for one image as a reference image.

  The decoded image storage unit 210 stores the input block data of the decoded image as new reference image data, and outputs the data to the intra prediction unit 211, the inter prediction unit 212, and the motion vector calculation unit 213.

  The intra prediction unit 211 generates block data of the predicted image from the already-encoded reference pixels for the processing target block of the encoding target image. The intra prediction unit 211 performs prediction using a plurality of prediction directions, and determines an optimal prediction direction. With respect to the prediction direction, the difference value is output to the entropy encoding unit 204 in order to include the difference value with the prediction direction of the encoded block in the bitstream.

  The inter prediction unit 212 performs motion compensation on the reference image data acquired from the decoded image storage unit 210 with the motion vector provided from the motion vector calculation unit 213. Thereby, block data as a motion-compensated reference image is generated. For the motion vector, the difference value with the motion vector (predicted vector) of the encoded block is output to the entropy encoding unit 204 in order to include the difference value in the bitstream.

  The motion vector calculation unit 213 has the configuration of the image processing apparatus 10 in the first embodiment, and uses the depth information, the block data in the encoding target image, and the reference image acquired from the decoded image storage unit 210 to perform motion. Find a vector.

  The motion vector calculation unit 213 outputs the obtained motion vector to the inter prediction unit 212 and outputs motion vector information including information indicating the reference image to the entropy encoding unit 204.

  The block data output from the intra prediction unit 211 and the inter prediction unit 212 is input to the predicted image selection unit 215.

  The predicted image selection unit 215 selects one of the block data acquired from the intra prediction unit 211 and the inter prediction unit 212 as a predicted image. The selected prediction image is output to the prediction error signal generation unit 201.

  Note that the configuration of the encoding device 20 illustrated in FIG. 5 is an example, and the configurations may be combined or the configurations may be appropriately changed as necessary.

  As described above, according to the second embodiment, it is possible to improve the image quality using the depth information at the time of image coding.

[Example 3]
FIG. 6 is a block diagram illustrating an example of a schematic configuration of the image processing apparatus 40 according to the third embodiment. An image processing apparatus 40 illustrated in FIG. 6 is an example of an image processing apparatus in which the image processing apparatus 10 and the encoding apparatus 20 described in the first and second embodiments are implemented by software.

  As shown in FIG. 6, the image processing apparatus 40 includes a control unit 401, a main storage unit 402, an auxiliary storage unit 403, a drive device 404, a network I / F unit 406, an input unit 407, and a display unit. 408. These components are connected to each other via a bus so as to be able to transmit and receive data.

  The control unit 401 is a CPU (Central Processing Unit) that controls each device, calculates data, and processes in a computer. The control unit 401 is an arithmetic device that executes an image processing program stored in the main storage unit 402 or the auxiliary storage unit 403. The control unit 401 receives data from the input unit 407 and the storage device, calculates and processes the data, and then outputs the data to the display unit 408 and the storage device.

  Further, the control unit 401 can implement the processing described in the first and second embodiments by executing an image processing program.

  The main storage unit 402 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. The main storage unit 402 is a storage device that stores or temporarily stores programs and data such as OS (Operating System) and application software that are basic software executed by the control unit 401.

  The auxiliary storage unit 403 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 404 reads the program from the recording medium 405, for example, a flexible disk, and installs it in the storage unit.

  A predetermined program is stored in the recording medium 405, and the program stored in the recording medium 405 is installed in the image processing apparatus 40 via the drive device 404. The installed predetermined program can be executed by the image processing apparatus 40.

  The network I / F unit 406 is a peripheral having a communication function connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) constructed by a data transmission path such as a wired and / or wireless line. This is an interface between the device and the image processing apparatus 40.

  The input unit 407 includes a keyboard having cursor keys, numeric input, various function keys, and the like, and a mouse and a slide pad for performing key selection on the display screen of the display unit 408. The display unit 408 is configured by an LCD (Liquid Crystal Display) or the like, and performs display according to display data input from the control unit 401.

  Each unit of the image processing apparatus 10 illustrated in FIG. 3 can be realized by, for example, the control unit 401 and the main storage unit 402 as a work memory.

  Further, the decoded image storage unit 210 illustrated in FIG. 5 is realized by, for example, the main storage unit 402 or the auxiliary storage unit 403. The configuration other than the decoded image storage unit 210 illustrated in FIG. 5 includes, for example, a control unit 401 and a work memory. This can be realized by the main storage unit 402.

  Further, the decoded information storage unit 305 and the frame memory 311 illustrated in FIG. 6 can be realized by the main storage unit 402 or the auxiliary storage unit 403, for example. The configuration other than the decoded information storage unit 305 and the frame memory 311 illustrated in FIG. 6 can be realized by the control unit 401 and the main storage unit 402 as a work memory, for example.

  The program executed by the image processing apparatus 40 has a module configuration including each unit described in the first and second embodiments. As actual hardware, when the control unit 401 reads out and executes a program from the auxiliary storage unit 403, one or more of the above-described units are loaded onto the main storage unit 402, and one or more of the respective units are main. It is generated on the storage unit 402.

  As described above, the image processing described in the first and second embodiments may be realized as a program for causing a computer to execute the image processing. The processing described in the first and second embodiments can be realized by installing this program from a server or the like and causing the computer to execute the program.

  It is also possible to record the program in the recording medium 405 and cause the processing unit such as a computer or a portable terminal to read the recording medium 405 on which the program is recorded, thereby realizing the above-described image processing.

  Note that the recording medium 405 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk, and information that is electrically stored such as a ROM or a flash memory. Various types of recording media such as a semiconductor memory for recording can be used.

  Further, the image processing described in each of the above embodiments may be implemented in one or a plurality of integrated circuits. Note that the image processing device 40 according to the third embodiment may have a function as at least one of the image processing device 10 and the encoding device 20 as described above.

  In addition, the image processing device 10, the encoding device 20, and the image processing device 40 in each of the embodiments described above include an encoding technique for detecting a motion vector using depth information, and bits encoded by this encoding technique. Applicable to decoding of streams. H.264 / AVC and H.264 It is not limited to 265 / HEVC.

  Each embodiment has been described in detail above. However, the present invention is not limited to the specific embodiment, and various modifications and changes other than the above-described modification are possible within the scope described in the claims. .

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 20 Encoding apparatus 40 Image processing apparatus 140 Motion vector candidate specific | specification part 150 Depth information clustering part 160 Object determination part 162 Acquisition part 164 Depth determination part 170 Motion vector determination part 171 1st determination part 172 2nd determination part 173 Third determination unit 174 Selection unit 175 Measurement unit 204 Entropy encoding unit 213 Motion vector calculation unit 301 Entropy decoding unit 401 Control unit 402 Main storage unit 403 Auxiliary storage unit

Claims (6)

An image processing apparatus that performs processing in units of blocks on an image including depth information,
A motion vector candidate that specifies a plurality of motion vector candidates of the processing target block based on the degree of coincidence between the processing target block that is one of the plurality of blocks of the reference frame and the image of the reference frame that is referred to A specific part,
The processing target block and the corresponding block of the reference frame belong to the same object by using the depth information of the processing target block and the depth information of the corresponding block of the reference frame for which the motion vector is obtained. An object determination unit for determining whether or not,
A motion vector determination unit that determines a motion vector of the processing target block based on the degree of coincidence corresponding to each of the plurality of motion vector candidates and the determination result of the object determination unit;
An image processing apparatus. A depth information clustering unit that performs clustering processing on the depth information of the base frame and the reference frame;
The object determination unit performs the determination using a result of the clustering.
The image processing apparatus according to claim 1. The motion vector determination unit
When there is a motion vector determined to belong to the same object as the block to be processed, the block of the reference frame related to the plurality of motion vector candidates, among the motion vectors determined to belong to the same object, The motion vector having the highest degree of coincidence is determined as the motion vector of the processing target block, or any block of the reference frame related to a plurality of motion vector candidates belongs to the same object as the processing target block. If it is determined not to, the motion vector having the highest degree of coincidence is determined as the motion vector of the processing target block.
The image processing apparatus according to claim 1 or 2. The motion vector determination unit
When it is determined that none of the reference frames related to a plurality of motion vector candidates belong to the same object as the processing target block, and the processing target block can be further divided into a plurality of sub-blocks The image processing apparatus according to claim 1, wherein an instruction to repeat the processing of the image processing apparatus is output with the sub-block as a processing target block.   An encoding device comprising the image processing device according to any one of claims 1 to 4.   An encoding program for causing a computer to function as the encoding device according to claim 5.

Images