WO2011129361A1 - 画像処理システム、画像処理方法、およびプログラム - Google Patents
画像処理システム、画像処理方法、およびプログラム Download PDFInfo
- Publication number
- WO2011129361A1 WO2011129361A1 PCT/JP2011/059160 JP2011059160W WO2011129361A1 WO 2011129361 A1 WO2011129361 A1 WO 2011129361A1 JP 2011059160 W JP2011059160 W JP 2011059160W WO 2011129361 A1 WO2011129361 A1 WO 2011129361A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- frame
- processing system
- frames
- unit
- image processing
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims description 161
- 238000003672 processing method Methods 0.000 title claims description 5
- 238000001514 detection method Methods 0.000 claims abstract description 217
- 238000000034 method Methods 0.000 claims abstract description 174
- 230000008569 process Effects 0.000 claims abstract description 137
- 238000004364 calculation method Methods 0.000 claims description 72
- 238000007906 compression Methods 0.000 claims description 65
- 230000006835 compression Effects 0.000 claims description 60
- 238000003384 imaging method Methods 0.000 claims description 41
- 238000011156 evaluation Methods 0.000 claims description 21
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 230000008685 targeting Effects 0.000 claims description 3
- 230000010365 information processing Effects 0.000 description 84
- 238000010586 diagram Methods 0.000 description 47
- 230000004048 modification Effects 0.000 description 23
- 238000012986 modification Methods 0.000 description 23
- 230000009467 reduction Effects 0.000 description 18
- 230000005540 biological transmission Effects 0.000 description 11
- 230000006837 decompression Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 238000005259 measurement Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000015572 biosynthetic process Effects 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 238000003786 synthesis reaction Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 238000013139 quantization Methods 0.000 description 3
- 230000000052 comparative effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013144 data compression Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
Definitions
- the present invention relates to an image processing technique related to detection of corresponding points.
- a stereo image having a moving image mode has a large amount of data
- MPEG Motion Picture Experts Group
- extension processing can be considered.
- problems such as a longer corresponding point detection time due to an increase in the amount of computation required for the decompression process occur.
- the present invention has been made in view of the above problems, and an object thereof is to provide a technique for reducing the amount of calculation related to detection of corresponding points for a compressed moving image.
- an image processing system includes a first frame having a reference frame and a prediction frame in which each pixel is indicated by motion information based on another frame.
- An acquisition unit that acquires a second compressed moving image, and each set of frames between the first compressed moving image and the second compressed moving image is included in the second compressed moving image.
- a calculation unit that performs detection processing for detecting corresponding points corresponding to the reference points of one frame included in the first compressed moving image from the frame. In the image processing system, the calculation unit performs the detection process on the set of the prediction frames using motion information indicating the set of the prediction frames.
- An image processing system is the image processing system according to the first aspect, wherein the calculation unit performs the detection processing on a set of the prediction frames as a set of the reference frames. This is performed based on the detection result of the detection process for the image and the motion information indicating the set of the prediction frames.
- An image processing system is the image processing system according to the second aspect, wherein a single reference point included in one prediction frame in the detection process related to a set of the prediction frames.
- a calculation unit that calculates an evaluation value related to the similarity with the one reference point, and a comparison between the evaluation value and a preset threshold value
- a determination unit for determining reliability related to the one corresponding point.
- An image processing system is the image processing system according to the first aspect, wherein the calculation unit is configured to detect the set of the reference frames and the detection process for the set of the reference frames.
- a first temporary corresponding point corresponding to one reference point included in one prediction frame is obtained from the other prediction frame on the basis of the detection result of the first frame and the motion information indicating the set of the prediction frames.
- 1 detection unit comparing the region including the one reference point in the one prediction frame with the region including the one temporary corresponding point in the other prediction frame, from the other prediction frame And a second detection unit that detects one corresponding point corresponding to one reference point.
- An image processing system is the image processing system according to the first aspect, wherein the calculation unit is configured to detect the set of the prediction frames with respect to the set of reference frames.
- a first temporary corresponding point corresponding to one reference point included in one prediction frame is obtained from the other prediction frame on the basis of the detection result of the first frame and the motion information indicating the set of the prediction frames.
- a second detection unit that detects one corresponding point corresponding to the one reference point from the other post-expansion frame with reference to a region including the one temporary corresponding point in the frame.
- An image processing system is the image processing system according to any one of the first to fifth aspects, wherein each reference frame includes a compression process for deleting information related to a frequency in a predetermined band.
- the frame is subjected to a calculation using the phase-only correlation method while performing weighting for reducing the frequency component of the predetermined band from the remaining band for the predetermined band, thereby targeting a set of frames.
- the detection process is performed.
- An image processing system is the image processing system according to any one of the first to fifth aspects, wherein each of the reference frames is subjected to compression processing in units of a predetermined size block.
- the calculation unit sets a first area including one reference point for the first frame of the first compressed moving image according to the block, and sets the second frame of the second compressed moving image.
- the second region is set according to the block, and the calculation using the phase-only correlation method for the first region and the second region is performed.
- the detection process for a pair with a frame is performed.
- An image processing system is the image processing system according to any one of the first to seventh aspects, wherein the calculation unit is configured to perform the above-described processing for the set of the reference frame and the prediction frame.
- the detection process is performed after frame conversion is performed so as to match the format of one of the reference frame and the predicted frame.
- An image processing system is the image processing system according to the eighth aspect, wherein the arithmetic unit expands the prediction frame for the set of the reference frame and the prediction frame. The detection processing for the reference frame and the predicted frame after expansion is performed.
- An image processing system is the image processing system according to the first aspect, wherein the set of prediction frames includes a first prediction frame and a second prediction frame, and the calculation unit includes: By comparing the motion information related to the reference region including one reference point in the first prediction frame and the motion information related to the comparison region including one temporary corresponding point in the second prediction frame, One corresponding point corresponding to the one reference point is detected from two prediction frames.
- An image processing system is the image processing system according to the tenth aspect, wherein the motion information related to the reference region includes a one-dimensional one-dimensional distribution of motion information related to one direction and the A two-dimensional other reference distribution of motion information in another direction different from the one direction, and the motion information in the comparison area is a two-dimensional one comparison distribution of motion information in the one direction.
- a two-dimensional other comparison distribution of motion information in the other direction, and the calculation unit includes voxel information including the one reference distribution and the other reference distribution, and the one comparison distribution and the other comparison distribution. The one corresponding point corresponding to the one reference point is detected from the second prediction frame by comparison with the included voxel information.
- An image processing system is the image processing system according to any one of the first to eleventh aspects, wherein the first moving image is obtained by a first series of imaging for a subject.
- a second compression unit that generates the second compressed moving image by applying a compression process to the second moving image.
- An image processing system is the image processing system according to any one of the first to seventh aspects, and generates the first compressed moving image by performing compression processing on the first moving image.
- a first compression unit that performs the compression process on the second moving image
- a second compression unit that generates the second compressed moving image by performing a compression process on the second moving image.
- Each set of frames between the image and the second moving image operates so as to be any one of the set of the reference frame and the set of the prediction frames.
- the image processing method obtains first and second compressed moving images each having (a) a reference frame and a prediction frame in which each pixel is indicated by motion information based on another frame. And (b) for each set of frames between the first compressed video and the second compressed video, from one frame included in the second compressed video, the first Performing a detection process of detecting corresponding points corresponding to each reference point of one frame included in one compressed moving image.
- the detection process for the set of the prediction frames is performed using motion information indicating the set of the prediction frames.
- the program according to the fifteenth aspect is executed by a control unit included in the image processing system, thereby causing the image processing system to function as the image processing system according to any one of the first to thirteenth aspects.
- the image processing system can reduce the amount of calculation related to detection of corresponding points for a compressed moving image.
- the amount of calculation required for detecting corresponding points related to the prediction frame is reduced.
- the corresponding points are easily detected by using the temporary corresponding points, the corresponding points are detected while maintaining the detection accuracy of the corresponding points. The amount of calculation related to is reduced.
- the amount of calculation related to detection of corresponding points is reduced by reducing unnecessary calculations.
- the image processing system it is possible to detect corresponding points for different sets of frames while suppressing an increase in the amount of calculation.
- the calculation amount for detecting the corresponding points is reduced.
- the amount of calculation related to detection of corresponding points for a compressed moving image is reduced by any of the image processing method according to the fourteenth aspect and the program according to the fifteenth aspect.
- FIG. 1 is a diagram illustrating an outline of an information processing system according to each embodiment and a third modification.
- FIG. 2 is a diagram illustrating a functional configuration of the information processing system according to the first and second embodiments.
- FIG. 3 is a diagram illustrating an aspect of the first and second compressed moving images.
- FIG. 4 is a diagram for explaining the principle of corresponding point detection processing according to the first embodiment.
- FIG. 5 is a schematic view illustrating the form of each frame.
- FIG. 6 is a schematic view illustrating the form of each first reduced frame.
- FIG. 7 is a schematic view illustrating the form of each second reduced frame.
- FIG. 8 is a schematic view illustrating the form of each third reduced frame.
- FIG. 9 is a schematic diagram illustrating a setting mode of reference points for each frame.
- FIG. 9 is a schematic diagram illustrating a setting mode of reference points for each frame.
- FIG. 10 is a diagram illustrating a setting example of the reference point and the processing target point in the third reduced frame.
- FIG. 11 is a schematic diagram illustrating an example of setting a window for the third reduced frame.
- FIG. 12 is a schematic diagram illustrating an example of setting a window for the second reduced frame.
- FIG. 13 is a schematic diagram illustrating an example of setting a window for the first reduced frame.
- FIG. 14 is a schematic diagram illustrating a setting example of a window for the first reduced frame.
- FIG. 15 is a schematic diagram showing a window setting example for each frame.
- FIG. 16 is a schematic diagram showing a window setting example for each frame.
- FIG. 17 is a diagram illustrating a functional configuration related to the I detection process using the POC method.
- FIG. 18 is a diagram illustrating a distribution of POC values.
- FIG. 19 is a diagram for explaining the principle of the P detection process.
- FIG. 20 is a diagram for explaining the principle of the P detection process.
- FIG. 21 is a diagram illustrating a functional configuration related to the P detection process using the POC method.
- FIG. 22 is a flowchart illustrating an operation flow of the information processing system.
- FIG. 23 is a flowchart showing an operation flow of the information processing system.
- FIG. 24 is a flowchart showing an operation flow of the information processing system.
- FIG. 25 is a diagram illustrating a functional configuration of the information processing system according to the second embodiment.
- FIG. 26 is a diagram for explaining the compression mode and the detection process according to the second embodiment.
- FIG. 27 is a flowchart illustrating an operation flow of the information processing system.
- FIG. 28 is a flowchart illustrating an operation flow of the information processing system.
- FIG. 29 is a diagram for explaining the principle of the P detection processing according to the third embodiment.
- FIG. 30 is a diagram for explaining the principle of the P detection processing according to the third embodiment.
- FIG. 31 is a diagram illustrating a functional configuration of an information processing system according to the fourth embodiment.
- FIG. 32 is a diagram for explaining a reliability determination method according to the fourth embodiment.
- FIG. 33 is a flowchart showing an operation flow of the information processing system.
- FIG. 34 is a diagram illustrating a functional configuration of an information processing system according to the fifth embodiment.
- FIG. 35 is a diagram for explaining the principle of P detection processing according to the fifth embodiment.
- FIG. 36 is a flowchart showing an operation flow of the information processing system.
- FIG. 37 is a diagram illustrating a distribution of weighting coefficients according to the first modification.
- FIG. 38 is a diagram illustrating a distribution of weighting coefficients according to the first modification.
- FIG. 39 is a schematic diagram illustrating a window setting example according to the second modification.
- FIG. 40 is a schematic diagram illustrating a window setting example according to the second modification.
- FIG. 41 is a diagram illustrating a functional configuration of an information processing system according to the third modification.
- FIG. 42 is a flowchart showing an operation flow of the information processing system.
- FIG. 43 is a diagram for explaining the principle of corresponding point detection processing according to the fourth modification.
- FIG. 44 is a diagram for explaining the principle of corresponding point detection processing according to the fourth modification.
- FIG. 1 is a diagram showing a schematic configuration of an information processing system 1 according to the first embodiment of the present invention.
- the information processing system 1 includes a stereo camera 2, communication lines 3a and 3b, and an information processing device 4.
- the stereo camera 2 and the information processing apparatus 4 are connected via communication lines 3a and 3b so that various signals can be transmitted and received.
- the stereo camera 2 includes first and second cameras 2a and 2b.
- Each of the cameras 2a and 2b has a configuration of a digital camera having an image sensor such as a CCD, for example, and receives the light from the subject and converts the luminance information distribution related to the subject by photoelectric conversion to the image data ( Image acquisition is also performed.
- Each camera 2a, 2b captures a temporal change in the subject by performing imaging (also referred to as a series of imaging) a predetermined number of times (for example, 60 times) every predetermined time (for example, 1 second). (Also referred to as a moving image).
- first and second cameras 2a and 2b are arranged apart from each other in a predetermined direction, and sequentially perform processing of imaging the same subject from different viewpoints at the same timing in the same period.
- Each set of images captured at the same timing by the first and second cameras 2a and 2b is a so-called stereo image.
- the first camera 2a generates a compressed moving image (also referred to as a first compressed moving image) Ma by performing moving image compression on the moving image (also referred to as a first moving image) obtained by imaging, and this first compression.
- the moving image Ma is transmitted to the information processing device 4 via the communication line 3a.
- the second camera 2b performs moving image compression on a moving image (also referred to as a second moving image) obtained by imaging to generate a compressed moving image (also referred to as a second compressed moving image) Mb, and this second compressed moving image.
- Mb is transmitted to the information processing apparatus 4 via the communication line 3b.
- the moving image compression it is possible to adopt, for example, those complying with various standards such as so-called MPEG4. In the following, it is assumed that moving image compression according to MPEG4 is employed.
- the first and second compressed moving images Ma and Mb both have N images (N is a natural number of 2 or more) images (also referred to as frames).
- N is a natural number of 2 or more images (also referred to as frames).
- data indicating each frame and each frame itself are collectively referred to as a frame.
- the coordinates of pixels capturing the same subject portion are the same in the Y direction.
- the communication lines 3a and 3b may be wired lines using cables that can transmit and receive various signals, or may be wireless lines using wireless communication or the like.
- the information processing apparatus 4 is configured by a personal computer (personal computer), for example, and includes an operation unit 401, a display unit 402, an interface (I / F) unit 403, a storage unit 404, an input / output unit 405, and a control unit 406. .
- a personal computer personal computer
- the operation unit 401 includes, for example, a mouse and a keyboard.
- the display unit 402 includes, for example, a liquid crystal display.
- the I / F unit 403 receives a signal from the stereo camera 2 and transmits a signal to the stereo camera 2.
- the storage unit 404 includes, for example, a hard disk and stores various data, a program PG, and the like.
- the input / output unit 405 includes a disk drive to which a storage medium 9 such as an optical disk can be mounted. The input / output unit 405 reads information from the storage medium 9 and outputs the information to the control unit 406 and stores data output from the control unit 406. Write to medium 9.
- the control unit 406 includes a CPU 406a that functions as a processor and a memory 406b that temporarily stores data, and comprehensively controls each unit of the information processing apparatus 4.
- various functions, various information processing, and the like are realized by reading and executing the program PG in the storage unit 404.
- the information processing apparatus 4 functions as an image processing apparatus and the information processing system 1 functions as an image processing system under the control of the control unit 406 according to the program PG.
- FIG. 2 is a diagram illustrating a functional configuration of the information processing system 1.
- the first camera 2a includes an imaging unit 21a, a compression unit 22a, and a transmission unit 23a
- the second camera 2b includes an imaging unit 21b, a compression unit 22b, and a transmission unit 23b.
- the imaging unit 21a and the imaging unit 21b have the same configuration
- the compression unit 22a and the compression unit 22b have the same configuration
- the transmission unit 23a and the transmission unit 23b have the same configuration. Therefore, for the purpose of avoiding repeated description, the imaging unit 21a, the compression unit 22a, and the transmission unit 23a will be mainly described.
- the imaging unit 21a includes an imaging element and a signal processing unit.
- the image pickup device includes, for example, a CCD
- the signal processing unit includes, for example, a CDS circuit, an AGC circuit, and an A / D conversion circuit.
- the analog signal output from the image sensor is sequentially subjected to noise reduction processing by the CDS circuit, level adjustment processing by the AGC circuit, and conversion to a digital signal by the A / D conversion circuit.
- the imaging unit 21a obtains a first moving image including a plurality of frames indicated by digital signals by a series of imaging (also referred to as a first series of imaging).
- the imaging unit 21b performs imaging while synchronizing with the imaging unit 21a. Specifically, the imaging unit 21b digitally performs a series of imaging (also referred to as a second series of imaging) on the same subject as the first series of imaging at the same timing as the first series of imaging. A second moving image including a plurality of frames indicated by the signal is obtained.
- the imaging timing in the imaging units 21a and 21b may be controlled according to a signal from the control unit 406, for example.
- the compression unit 22a performs moving image compression on the first moving image to generate the first compressed moving image Ma. Specifically, the compression unit 22a performs moving picture compression according to MPEG4 including various processes such as discrete cosine transform (DCT) and quantization. In DCT and quantization, data compression processing is performed while an image area (also referred to as a block) of a predetermined size is used as one processing unit, and information relating to a frequency in a predetermined band (mainly a high frequency band) is deleted.
- the Each block having a predetermined size may be a square image region or the like having a predetermined number (8 or 16) of pixels on one side. Note that, similarly to the compression process 22a, the compression unit 22b performs moving image compression on the second moving image to generate the second compressed moving image Mb.
- one or more frames of a plurality of frames included in a moving image are set as reference frames (also referred to as reference frames or I-pictures), and other frames than the reference frames are based on the reference frames.
- a predicted frame in which each pixel is indicated by the motion information (also referred to as a predicted frame).
- the motion information relating to each pixel is the change in the coordinates of the pixel that captures each part of the subject between the i-1th frame (i is a natural number equal to or less than N) and the ith frame constituting the moving image. Indicating each pixel of the i-th frame.
- the reference frame has a pixel value (such as a luminance value) for each pixel
- the prediction frame has motion information corresponding to each pixel.
- the positions of the pixels constituting each frame are indicated by coordinates (X coordinate value and Y coordinate value) relating to two orthogonal XY axes, and a change in coordinates corresponding to motion information is represented by X What is necessary is just to express with the format of the motion vector shown by the value of a coordinate, and the value of a Y coordinate. That is, the motion information includes motion information in the X direction and motion information in the Y direction.
- the predicted frame is a frame in which each pixel is indicated by motion information obtained from a comparison between the i-th frame and the (i ⁇ 1) -th frame (also called a P picture). Or a frame (also referred to as a B picture) in which each pixel is indicated by motion information obtained from a comparison between the i-th frame and the (i ⁇ 1) -th and i + 1-th frames. Also good.
- the prediction frame is a P picture.
- FIG. 3 is a diagram illustrating an aspect of the first and second compressed moving images Ma and Mb.
- the i-th frame constituting the first compressed moving image Ma is described as a frame Mai related to imaging at the time Ti
- the i-th frame constituting the second compressed moving image Mb relates to imaging at the time Ti. It is described as a frame Mbi.
- the first compressed moving image Ma has N frames Ma1 to MaN.
- the first, fifth, and ninth frames Ma1, Ma5, and Ma9 are I pictures, and the second to fourth, sixth to eighth, and tenth frames Ma2 to Ma4, Ma6 to Ma8, Ma10, and MaN are included.
- the second compressed moving image Mb has N frames Mb1 to MbN.
- the third and seventh frames Mb3 and Mb7 are I pictures, and 1, 2, 4 to 6, 8 to 10, and the Nth frames Mb1, Mb2, Mb4 to Mb6, Mb8 to Mb10, MaN , P picture.
- each pixel is indicated by motion information based on the immediately preceding I picture. Also, in the P picture immediately after the P picture, each pixel is indicated by motion information based on the I picture because the pixel is indicated by motion information based on the immediately preceding P picture. Yes. It should be noted that all P pictures may have their pixels directly indicated by motion information based on the I picture.
- the transmitting unit 23a transmits the first compressed moving image Ma to the information processing device 4.
- the transmission unit 23 b transmits the second compressed moving image Mb to the information processing device 4.
- the information processing apparatus 4 as an image processing apparatus includes receiving units 41 a and 41 b and a calculation unit 42.
- the receiving units 41a and 41b are realized by an I / F unit 403 as an acquiring unit.
- the receiving unit 41a acquires the first compressed moving image Ma by receiving the first compressed moving image Ma output from the first camera 2a.
- the receiving unit 41b acquires the second compressed moving image Mb by receiving the second compressed moving image Mb output from the second camera 2b.
- the calculation unit 42 is realized by the program PG being executed by the control unit 406, and includes an expansion unit 421, a search unit 422, and a distance measurement unit 423 as a functional configuration.
- the decompression unit 421 decompresses the P pictures included in the first and second compressed moving images Ma and Mb as necessary, and after decompression, has pixel value information for each pixel as in the I picture.
- a P picture (also referred to as a D picture) is generated and output to the search unit 422.
- a D picture having the same form as an I picture corresponds to a reference frame. Further, the decompression unit 421 outputs the I picture and the P picture that does not require decompression to the search unit 422 as they are.
- the first set of frames Ma1, Mb1, the third set of frames Ma3, Mb3, the fifth set of frames Ma5, Mb5, and 7 The ninth set of frames Ma7 and Mb7 and the ninth set of frames Ma9 and Mb9 are a set of an I picture and a P picture, respectively.
- the expansion unit 421 expands the P pictures Mb1, Ma3, Mb5, Ma7, and Mb9 to generate D pictures Md1, Md3, Md5, Md7, and Md9.
- the first and second compressed moving images Ma and Mb may include a frame that is a pair of I pictures.
- the first and second compressed moving images Ma and Mb become the first and second compressed moving images Maa and Mbb by converting some P pictures into D pictures by the decompression unit 421.
- each frame included in the first compressed moving image Maa is also referred to as “one frame”
- each frame included in the second compressed moving image Mbb is also referred to as “the other frame”.
- the search unit 422 detects a correspondence relationship between the first compressed moving image Maa and the second compressed moving image Mbb for pixels that capture the same portion of the subject for each set of frames related to imaging at the same time. To do.
- detection reference points also referred to as reference points
- the second compressed moving image Mbb is set for each reference point.
- a process (also referred to as a detection process) is performed in which a point corresponding to the reference point (also referred to as a corresponding point) is detected from the other frame included in.
- This detection process includes a detection process for a pair of I pictures and a pair of an I picture and a D picture (also referred to as an I detection process), and a detection process for a pair of P pictures (P detection process). Is also included). That is, in the calculation unit 42, if the target of the corresponding point detection process is a pair of an I picture and a P picture, the decompression unit 421 decompresses the P picture into a D picture, and then creates the I picture and the D picture. I detection processing for the above is performed. If the target of the corresponding point detection process is a pair of P pictures, the P detection process is performed without decompressing the P picture.
- a set of frames whose resolution is reduced to a predetermined number (for example, three) based on each set of frames is generated.
- the relationship between the reference point and the corresponding point, which captures the same part of the subject, is detected in order from the set of frames with the lowest.
- the detection result for a set of frames whose resolution is one step lower is used for the next detection for a set of frames whose resolution is one step higher.
- FIG. 5 is a schematic view illustrating the form of each frame included in the first and second compressed moving images Maa and Mbb.
- a large number of pixels are arranged in a matrix.
- a first predetermined number (here, 480) of pixels are arranged in the vertical direction (Y direction)
- a second predetermined number (here, 640) of pixels are arranged in the horizontal direction (X direction).
- the upper left position is the origin
- the position in the horizontal direction of each pixel constituting each frame is indicated by the X coordinate
- the position in the vertical direction is indicated by the Y coordinate.
- each pixel in each frame, the position of each pixel is indicated by coordinates (X, Y). For example, if one pixel is shifted in the right direction (X direction), the value of the X coordinate increases by one, and the downward direction (Y direction). If the pixel is shifted by one pixel, the value of the Y coordinate increases by one.
- FIG. 6 is a diagram illustrating a form of a frame (also referred to as a first reduced frame) generated by reducing the resolution of each frame by one step.
- One-step resolution reduction is performed by, for example, 1 out of a plurality of vertical pixel columns (also referred to as vertical lines) and a plurality of horizontal pixel rows (also referred to as horizontal lines) constituting each frame. This is realized by thinning out vertical lines every other column and horizontal lines every other row.
- the first reduced frame has a configuration in which 240 pixels in the vertical direction and 320 pixels in the horizontal direction are arranged in a matrix.
- a first reduced frame (also referred to as a first reduced frame) Ma1 S1 is generated from one frame Ma1
- a first reduced frame (also referred to as the other first reduced frame) Md1 S1 is generated from the other frame Md1. Is done. Further, while the first reduction after frame Ma2 S1 whereas the frame Ma2 is generated, the other first after the reduction frame Mb2 S1 is generated from the other frame Mb2.
- FIG. 7 is a diagram illustrating a form of a frame (also referred to as a second reduced frame) generated by reducing the resolution of each frame by two stages.
- a frame also referred to as a second reduced frame
- two-stage resolution reduction is performed by thinning out every other vertical line and every other horizontal line among the plurality of vertical lines and horizontal lines constituting each first reduced frame. It is realized by being.
- the second reduced frame has a configuration in which 120 pixels in the vertical direction and 160 pixels in the horizontal direction are arranged in a matrix.
- a second reduced frame (also referred to as a second reduced frame) Ma1 S2 is generated from one frame Ma1
- a second reduced frame (the other second reduced frame) Md1 S2 is generated from the other frame Md1. Is done.
- the frame Ma2 second reduced after frame Ma2 S2 is generated, and the other second reduction after frame Mb2 S2 is generated from the other frame Mb2.
- FIG. 8 is a diagram illustrating a form of a frame (also referred to as a third reduced frame) generated by reducing the resolution of each frame by three stages.
- the three-stage resolution reduction can be achieved by, for example, thinning out every other vertical line and every other horizontal line among a plurality of vertical lines and horizontal lines constituting each second reduced frame. It is realized by being.
- the third reduced frame has a configuration in which 60 pixels in the vertical direction and 80 pixels in the horizontal direction are arranged in a matrix.
- a third reduced frame (also referred to as a third reduced frame) Ma1 S3 is generated from one frame Ma1
- a third reduced frame (also referred to as the third reduced frame) Md1 S3 is generated from the other frame Md1. Is done. Also, whereas the third reduction after frame Ma2 S3 whereas a frame Ma2 is generated, and the other third reduction after frame Mb2 S3 is generated from the other frame Mb2.
- a point (also referred to as a reference point) Sp1 which is a reference for detecting a corresponding point, is set for one frame.
- the upper left pixel of one frame is the starting point, and as shown by the arrow in FIG. 9, the reference point Sp1 is shifted by one pixel from the left end to the right end along one horizontal line with respect to the one frame.
- the reference point Sp1 is shifted one pixel at a time from the left end to the right end along the one horizontal line (+ Y side). The operations set in order are repeated. Further, here, every time a corresponding point corresponding to one reference point Sp1 is detected on the other frame, the next reference point Sp1 is set.
- the initial parallax serving as a reference for detecting corresponding points for a set of third reduced frames is temporarily set.
- the initial parallax is set to, for example, zero.
- the point (reference point) Sp1 S3 corresponding to the reference point Sp1 is set to the third reduced frame (for example, frames Ma1 S3 and Ma2 S3 ).
- the other third reduction rear frame e.g., frame Md1 S3, Mb2 S3
- the predetermined initial parallax is a shift amount of the X coordinate between the position of the reference point Sp1 S3 on the one third reduced frame and the position of the processing target point Pp1 S3 on the other third reduced frame.
- one third reduced frame for example, one third reduced frame Ma1 S3 , Ma2 S3
- other third reduction rear frame e.g., the other third reduction after frame Md1 S3, Mb2 S3 window including around the target point Pp1 S3 in Wb S3 is set.
- the reference region Wa S3 and the comparison region Wb S3 are regions having the same size and the same shape (here, squares), and are configured, for example, by arranging 17 pixels in the vertical direction and the horizontal direction, respectively.
- C3 A phase-only correlation method (POC method) described later is used to calculate the correlation between the reference region Wa S3 and the comparison region Wb S3, and the corresponding point is detected on the third reduced frame.
- POC method phase-only correlation method
- a reference parallax (reference parallax) is temporarily set for detection of corresponding points for a set of second reduced frames.
- This reference parallax is set based on the detection result in the step (C3). Specifically, this reference parallax is obtained by calculating the X coordinate between the position of the reference point Sp1 S3 in the third reduced frame and the position of the corresponding point in the other third reduced frame detected in step (C3). Corresponds to the amount of deviation.
- the reference point Sp1 S2 corresponding to the reference point Sp1 is set to the second reduced frame (for example, frames Ma1 S2 and Ma2 S2 ).
- the other second reduction rear frame e.g., frame Md1 S2, Mb2 S2
- the target point Pp1 S2 corresponding to the detected corresponding points in step (C3) is set.
- (C5) as illustrated in Figure 12 (also referred to as the reference area) includes a window around a reference point Sp1 S2 while in the second after the reduction frame (e.g. while the second reduction after frame Ma1 S2, Ma2 S2) Wa S2 Is set. Further, (also referred to as comparative region) while the second reduction rear frame (e.g., while the second reduction after frame Md1 S2, Mb2 S2) window including around the target point Pp1 S2 in Wb S2 is set. Note that the reference area Wa S2 and the comparison area Wb S2 have the same shape and the same size as the reference area Wa S3 and the comparison area Wb S3 .
- the POC method is used to calculate the correlation between the reference area Wa S2 and the comparison area Wb S2, and the corresponding points are detected on the second reduced frame.
- a reference parallax is temporarily set for a set of first reduced frames.
- This reference parallax is set based on the detection result in the step (C6). Specifically, this reference parallax is obtained by calculating the X coordinate between the position of the reference point Sp1 S2 on the one second reduced frame and the position of the corresponding point on the other second reduced frame detected in step (C6). Corresponds to the amount of deviation.
- the reference point Sp1 S1 corresponding to the reference point Sp1 is set to the first reduced frame (for example, frames Ma1 S1 and Ma2 S1 ).
- the processing target point Pp1 S1 corresponding to the corresponding point detected in the step (C6) is set in the other first reduced frame (for example, the frames Md1 S1 and Mb2 S1 ).
- a window (also referred to as a reference area) Wa S1 that includes the reference point Sp1 S1 as the center in one first reduced frame (for example, one first reduced frame Ma1 S1 , Ma2 S1 ). Is set.
- the reference area Wa S1 and the comparison area Wb S1 are areas having the same shape and the same size as the reference areas Wa S2 and Wa S3 and the comparison areas Wb S2 and Wb S3 .
- the POC method is used to calculate the correlation between the reference area Wa S1 and the comparison area Wb S1, and the corresponding point is detected on the first reduced frame.
- a reference parallax is temporarily set for a set of frames.
- This reference parallax is set based on the detection result in the step (C9). Specifically, this reference parallax is obtained by calculating the X coordinate between the position of the reference point Sp1 S2 in the first reduced frame and the position of the corresponding point in the other first reduced frame detected in step (C9). Corresponds to the amount of deviation.
- the reference point Sp1 is set to one frame (for example, frames Ma1 and Ma2).
- the processing target point Pp1 corresponding to the corresponding point detected in the step (C9) is set in the other frame (for example, the frames Md1 and Mb2).
- a window (also referred to as a reference region) Wa including the reference point Sp1 as a center is set in one frame (for example, one frame Ma1, Ma2).
- a window (also referred to as a comparison area) Wb including the processing target point Pp1 as a center is set in the other frame (for example, the other frame Md1, Mb2).
- the reference region Wa and the comparison region Wb are regions having the same shape and the same size as the reference regions Wa S1 , Wa S2 , Wa S3 and the comparison regions Wb S1 , Wb S2 , Wb S3 .
- the distance measurement unit 423 calculates the parallax ⁇ d obtained from the coordinates of the reference point detected by the search unit 422 and the corresponding point using the principle of triangulation for each reference point for each set of frames. Based on this, the distance D from the stereo camera 2 (that is, the viewpoint) to the subject is derived.
- the separation distance B and the focal distance f are uniquely set by design.
- Information on the distance D related to each reference point derived by the distance measuring unit 423 is output from the calculation unit 42 and stored in the storage unit 404 or the like.
- the corresponding point detection method using the POC method executed by the search unit 422 includes a detection method related to the I detection process and a detection method related to the P detection process.
- FIG. 17 is a diagram for specifically explaining a corresponding point detection method using the POC method according to the I detection process.
- a method for detecting corresponding points between one frame and the other frame will be described as an example.
- a window (reference area) Wa setting process T1a for one frame and a window (comparison area) Wb setting process T1b for the other frame are performed.
- image areas corresponding to the reference area Wa and the comparison area Wb are expressed by the following equation (1).
- N 1 and N 2 are substituted for the subscript P of W in the proviso of Equation 2 , and 1 and 2 are substituted for the subscript s of k.
- a synthesis process T4 using an arithmetic expression expressed by the following expression 4 is performed, and a two-dimensional inverse Fourier transform process T5 using an arithmetic expression expressed by the following expression 5 is performed. Done. Thereby, the correlation calculation between the images is performed, and the result (POC value) is output.
- the POC value at a location with high correlation in the window (N 1 ⁇ N 2 ) is large, and the position corresponding to the peak Jc of the POC value in the comparison region Wb on the frame is On the other hand, it corresponds to the corresponding point on the other frame corresponding to the center point (reference point) Sp1 of the reference area Wa on the frame. Therefore, by detecting the peak Jc of the POC value, the corresponding point on the other frame corresponding to the reference point Sp1 on the one frame is detected.
- the amplitude component of the image is removed, and the correlation calculation is performed using only the phase component of the image. Is detected with high accuracy.
- the P picture has, for each pixel, motion information in the X direction (also referred to as motion information X) and motion information in the Y direction (also referred to as motion information Y). That is, the P picture has a distribution of motion information X and a distribution of motion information Y. Therefore, as shown in FIG. 19, the reference area Wa set in the P picture has a two-dimensional distribution (also referred to as X distribution or one reference distribution) of motion information X related to the reference area Wa, And a two-dimensional distribution (also referred to as Y distribution or the other reference distribution) of motion information Y related to the reference area Wa. Accordingly, as shown in FIG.
- data relating to the reference region Wa can be handled as a rectangular parallelepiped image region (also referred to as voxel information) Vxa formed by stacking the X distribution Wau and the Y distribution Wav.
- Vxa rectangular parallelepiped image region
- the direction in which the X distribution Wau and the Y distribution Wav are stacked is the L direction perpendicular to the XY plane.
- the comparison area Wb set in the P picture is compared with the two-dimensional distribution (also referred to as X distribution or one comparison distribution) Wbu of the motion information X related to the comparison area Wb.
- a two-dimensional distribution also referred to as Y distribution or other comparison distribution
- the data related to the comparison region Wb can be handled as a rectangular parallelepiped image region (also referred to as voxel information) Vxb formed by stacking the X distribution Wbu and the Y distribution Wbv in the L direction. It is.
- FIG. 21 is a diagram for specifically explaining how to obtain the distribution of POC values in the P detection process.
- voxel information Vxa and the voxel information Vxb a predetermined number N 1 pixels are arranged along the X direction, a predetermined number N 2 pixels are arranged along the Y direction, and a predetermined number along the L direction. It is treated as a rectangular parallelepiped image region (voxel information) in which N 3 (here, two) pixels are arranged. These voxel information are expressed by the following equation (6).
- N 1 , N 2 , and N 3 are substituted for the subscript P of W in the proviso of Equation 7 above, and 1 , 2 , and 3 are substituted for the subscript s of k.
- normalization processing TV3a and TV3b for removing the amplitude component of the image is obtained using an arithmetic expression expressed by the following equation (8). Done.
- the size of the windows (reference area and comparison area) (N 1 and N 2 ) is set smaller in the P detection process than in the I detection process. For example, N 2 may be halved.
- One set of the first reduced frame and the other first reduced frame, one set of the second reduced frame and the other second reduced frame, and one third reduced frame and the other third reduced frame can be realized by the same method as the detection of the corresponding points according to the pair of the one frame and the other frame described above.
- the data related to the reference region Wa S1 can be handled as voxel information Vxa S1 of a rectangular parallelepiped in which the X distribution Wau S1 and the Y distribution Wav S1 are stacked in the L direction, and the data related to the comparison region Wb S1 is also X
- the distribution Wbu S1 and the Y distribution Wbv S1 can be handled as voxel information Vxb S1 of a rectangular parallelepiped stacked in the L direction.
- the data related to the reference area Wa S2 can be handled as voxel information Vxa S2 of a rectangular parallelepiped in which the X distribution Wau S2 and the Y distribution Wav S2 are stacked in the L direction, and the data related to the comparison area Wb S2 is also X
- the distribution Wbu S2 and the Y distribution Wbv S2 can be handled as voxel information Vxb S2 of a rectangular parallelepiped stacked in the L direction.
- the data related to the reference region Wa S3 can be handled as voxel information Vxa S3 of a rectangular parallelepiped in which the X distribution Wau S3 and the Y distribution Wav S3 are stacked in the L direction, and the data related to the comparison region Wb S3 is also X
- the distribution Wbu S3 and the Y distribution Wbv S3 can be handled as voxel information Vxb S3 of a rectangular parallelepiped stacked in the L direction.
- step S1 the first and second moving images are acquired by imaging by the imaging units 21a and 21b.
- step S2 the compression units 22a and 22b perform moving image compression on the first and second moving images to generate the first and second compressed moving images Ma and Mb, which are output to the transmission units 23a and 23b.
- step S3 the first and second compressed moving images Ma and Mb are transmitted to the information processing apparatus 4 by the transmission units 23a and 23b.
- step S4 the first and second compressed moving images Ma and Mb are received by the receiving units 41a and 41b and output to the decompressing unit 421.
- step S5 the decompression unit 421 decompresses the P pictures included in the first and second compressed moving images Ma and Mb as necessary to generate a D picture.
- step S5 the first and second compressed moving images Maa and Mbb are generated by performing the operation flow shown in FIG.
- step S51 a numerical value i indicating the order of a set of frames to be processed among the first and second compressed moving images Ma and Mb is set to 1.
- step S52 the i-th set of frames of the first and second compressed moving images Ma and Mb is set as a processing target.
- step S53 it is determined whether or not the i-th set of frames is an I-picture set. If the i-th set of frames is not a set of I pictures, the process proceeds to step S54. If the i-th set of frames is a set of I pictures, the process proceeds to step S55.
- step S54 it is determined whether or not the i-th set of frames is a set of P pictures. If the i-th set of frames is a P picture set, the process proceeds to step S55. If the i-th set of frames is not a P picture set, the process proceeds to step S56.
- step S55 expansion of the i-th set of frames is prohibited.
- the i-th set of frames is output to the search unit 422 as it is without the i-th set of frames being expanded at all by the expansion unit 421.
- step S56 the P picture included in the i-th set of frames is expanded.
- step S57 it is determined whether or not the numerical value i is N.
- N indicates the number of frames constituting each of the first and second compressed moving images Ma and Mb.
- the process proceeds to step S58, the numerical value i is incremented by 1, and the process proceeds to step S52.
- the numerical value i is N, the operation flow ends.
- step S6 of FIG. 22 the search unit 422 detects corresponding points.
- step S6 the operation flow shown in FIG. 24 is performed.
- step S61 a numerical value i indicating the order of a set of frames to be processed among the first and second compressed moving images Maa and Mbb is set to 1.
- step S62 a set of i-th frames of the first and second compressed moving images Maa and Mbb is set as a processing target.
- step S63 it is determined whether or not the i-th set of frames is a set of I pictures or a set of I pictures and D pictures. If the i-th frame set is an I picture set or an I picture and D picture set, the process proceeds to step S64. If the i-th frame set is not an I picture set or an I picture and D picture set, the process proceeds to step S65. move on.
- step S64 an I detection process is performed.
- step S65 P detection processing is performed.
- step S66 it is determined whether or not the numerical value i is N.
- N indicates the number of frames constituting each of the first and second compressed moving images Maa and Mbb. If the numerical value i is not N, the process proceeds to step S67, where the numerical value i is incremented by 1, and the process proceeds to step S62. On the other hand, if the numerical value i is N, the operation flow ends.
- the distance measurement unit 423 determines the set of frames included in the first and second compressed moving images Maa and Mbb from the subject to the subject based on the corresponding points detected in step S6. A distance D is derived.
- FIG. 25 is a diagram illustrating a functional configuration of the information processing system 1A.
- the stereo camera 2 is changed to a stereo camera 2A in which video compression in the compression units 22a and 22b is synchronized, and the calculation unit 42 is expanded.
- the part 421 is removed, and the search part 422 is changed to a calculation part 42A replaced with the search part 422A.
- the compression units 22a and 22b are configured so that each set of frames between the first moving image and the second moving image is a set of either one set of I pictures or a set of P pictures.
- the first and second compressed moving images Ma and Ma are generated by operating so that Note that the synchronized moving image compression operation in the compression units 22a and 22b can be realized by a control unit or a control unit 406 built in the stereo camera 2A.
- FIG. 26 illustrates the first and second compressed moving images Ma and Mb generated by the stereo camera 2A.
- a set of frames every four frames such as a first set of frames Ma1, Mb1, a fifth set of frames Ma5, Mb5, and a ninth set of frames Ma9, Mb9 is a set of I pictures.
- each set of remaining frames is a set of P pictures.
- each set of frames is a set of I pictures or a set of P pictures. For this reason, the computing unit 42A does not need to expand the P picture.
- the functions of the calculation unit 42A (specifically, the search unit 422A and the distance measurement unit 423) are realized by the control unit 406 executing the program PGA stored in the storage unit 404.
- the detection process performed by the search unit 422A includes an I detection process and a P detection process.
- the operation flow shown in FIG. 27 is the operation flow in the information processing system 1 according to the first embodiment shown in FIG. 22, in which the process of step S2 is replaced with step S2A, and the process of step S5 is removed. , Step S6 is replaced with Step S6A. However, in step S4, the first and second compressed moving images Ma and Mb are output from the receivers 41a and 41b to the search unit 422A.
- step SA2 the first and second moving images are compressed by the compression units 22a and 22b to generate the first and second compressed moving images Ma and Mb, which are output to the transmission units 23a and 23b.
- each set of frames included in the first and second compressed moving images Ma and Mb is a set of I pictures or a set of P pictures.
- step SA6 the search unit 422A performs corresponding point detection processing.
- step SA6 the operation flow shown in FIG. 28 is performed.
- step SA61 a numerical value i indicating the order of a set of frames to be processed among the first and second compressed moving images Ma and Mb is set to 1.
- step SA62 a set of i-th frames of the first and second compressed moving images Ma and Mb is set as a processing target.
- step SA63 it is determined whether or not the set of the i-th frame is a set of I pictures. If the i-th set of frames is a set of I pictures, the process proceeds to step SA64. If the set of i-th frames is not a set of I pictures, the process proceeds to step SA65.
- step SA64 I detection processing is performed.
- step SA65 P detection processing is performed.
- step SA66 it is determined whether or not the numerical value i is N.
- N indicates the number of frames constituting the first and second compressed moving images Ma and Mb, respectively. If the numerical value i is not N, the process proceeds to step SA67, where the numerical value i is incremented by 1, and the process proceeds to step SA62. On the other hand, if the numerical value i is N, the operation flow ends.
- the information processing system 1B is based on the information processing system 1 according to the first embodiment, and the calculation unit 42 is changed to a calculation unit 42B in which the search unit 422 is replaced with the search unit 422B. It has been done. Note that the functions of the calculation unit 42B (specifically, the expansion unit 421, the search unit 422B, and the distance measurement unit 423) are realized by the program PGB stored in the storage unit 404 being executed by the control unit 406.
- the calculation unit 42B specifically, the expansion unit 421, the search unit 422B, and the distance measurement unit 423
- the P detection processing for a set of P pictures is performed by detecting the detection result of the I detection processing for a set of I pictures or a set of I pictures and D pictures, and a set of P pictures. This is performed based on motion information included in the picture. In other words, for the pair of P pictures, the reference point and the corresponding point corresponding to the reference point are indirectly obtained based on the detection result of the I detection process and the motion information.
- FIG. 29 and 30 are diagrams for explaining the principle of the P detection process according to this embodiment.
- FIG. 29 shows an example of the detection result of the I detection process.
- a corresponding point Pd1 X coordinate is Xd1, Y coordinate is Yd1 corresponding to the reference point Pa1 (X coordinate is Xa1, Y coordinate is Ya1) is shown. Yes.
- the search unit 422B searches for the first compressed video Maa based on motion information included in the P picture Ma2, which is the next frame of the I picture Ma1, out of the P pictures Ma2.
- a reference point Pa2 corresponding to the reference point Pa1 is derived. For example, if the motion information of the reference point Pa2 has the form of a motion vector indicated by the X coordinate value ua and the Y coordinate value va, the X coordinate value of the reference point Pa2 is the value of the reference point Pa1.
- the X coordinate value Xa1 is added to the X coordinate value ua of the motion information to obtain Xa1 + ua
- the Y coordinate value of the reference point Pa2 is the Y coordinate value Ya1 of the reference point Pa1
- the Y coordinate value va of the motion information is The added Ya1 + va.
- the search unit 422B determines, for the second compressed moving image Mbb, D among the P pictures Mb2 based on the motion information included in the P picture Mb2, which is the next frame of the D picture Md1.
- a corresponding point Pb2 corresponding to the corresponding point Pd1 of the picture Md1 is derived. For example, if the motion information of the corresponding point Pb2 has the form of a motion vector indicated by the X coordinate value ub and the Y coordinate value vb, the X coordinate value of the corresponding point Pb2 is the value of the corresponding point Pd1.
- the X coordinate value Xd1 is added to the X coordinate value ub of the motion information to obtain Xd1 + ub, the Y coordinate value of the corresponding point Pb2 is Yd, the Y coordinate value Yd1 of the corresponding point Pd1 is the Y coordinate value vb of the motion information.
- the calculation unit 42 performs P detection processing related to the set of P pictures of the next frame. This is performed based on the detection result of the P detection process for the pair of pictures and the motion information included in the pair of P pictures in the next frame.
- the P detection processing related to each pair of P pictures is performed as a set of I pictures and an I picture and a D picture. Based on the detection result of the I detection process for the most immediately preceding pair and the motion information from the most immediately preceding set to the set of P pictures that are the targets of the P detection process. Done. That is, the P detection process for each set of P pictures is performed by detecting the detection result of the I detection process for a set of I pictures or a set of I pictures and D pictures, and motion information included in the P pictures. Based on.
- step S65 step S6 shown in FIG. 22 is replaced with step SB6 in which the content of the corresponding point detection process is changed.
- This operation flow is realized by the control unit 406 reading and executing the program PGB in the storage unit 404.
- corresponding points are detected for a set of P pictures based on the result of the I detection process and the motion information of the P picture. For this reason, the amount of computation required for the P detection process is significantly reduced, and the amount of computation required for detecting the corresponding points related to the P picture is greatly reduced.
- FIG. 31 is a diagram illustrating a functional configuration of the information processing system 1C.
- a calculation unit 42B is added with an evaluation value calculation unit 424 and a reliability determination unit 425, and a distance measurement unit 423 performs distance measurement.
- the operation unit 42C is replaced with the unit 423C.
- the evaluation value calculation unit 424 is a value for evaluating the similarity between the reference point and the corresponding point with respect to the set of the reference point and the corresponding point obtained for the set of P pictures in the P detection process in the search unit 422B. (Also called evaluation value) is calculated.
- the degree of similarity is the degree of matching and indicates the high possibility of capturing the same subject portion.
- the evaluation value includes an area R1 including a reference point as a center in one P picture (also referred to as one evaluation area) and a corresponding point as a center in the other P picture.
- the value indicating the correlation related to the corresponding point may be used.
- a value indicating this correlation may be obtained from a distribution of POC values calculated by a process using the POC method similar to the P detection process, for example.
- the one evaluation region R1 and the other evaluation region R2 may be regions having the same size and the same shape (here, squares).
- the reliability determination unit 425 compares the evaluation value calculated by the evaluation value calculation unit 424 with the preset threshold value, and the reliability related to the set of the reference point and the corresponding point that is the target of the evaluation value calculation. Determine.
- the reliability indicates a certainty as a corresponding point with respect to the reference point. For example, if the evaluation value is greater than or equal to the threshold value, it is determined that the reliability of the pair of the reference point and the corresponding point is high, and if the evaluation value is less than the threshold value, the reliability of the pair of the reference point and the corresponding point is determined. A mode in which it is determined that the property is low is conceivable.
- the threshold may be stored in advance in the storage unit 404, for example.
- the distance measuring unit 423C derives the distance D from the stereo camera 2 to the subject based on the parallax ⁇ d obtained from the coordinates of the reference point and the corresponding point according to the determination result by the reliability determining unit 425. Specifically, for example, the distance D is not obtained for each pair of the reference point and the corresponding point determined as having low reliability by the reliability determination unit 425, and the reliability is determined by the reliability determination unit 425. For each set of reference points and corresponding points determined to be high, a mode in which the distance D is obtained can be considered. Thereby, reduction of the amount of calculation and ensuring of the calculation accuracy of distance D are realized.
- the method for deriving the distance D may be the same as the method for deriving by the distance measuring unit 423 according to the first to third embodiments, for example.
- FIG. 33 is a flowchart showing an operation flow in the information processing system 1C. This operation flow is realized by the control unit 406 reading and executing the program PGC in the storage unit 404. In this operation flow, first, the same processing as steps S1 to S5 and SB6 according to the third embodiment shown in FIG. 22 is performed, and then the processing of steps SC61, SC62, and SC7 is sequentially performed.
- step SC61 the evaluation value calculation unit 424 calculates an evaluation value for each set of reference points and corresponding points obtained in the P detection process in step SB6.
- step SC62 the reliability determination unit 425 compares the evaluation value calculated in step SC61 with the threshold value for each pair of the reference point and the corresponding point obtained in the P detection process of step SB6, and determines the reliability. Is determined.
- step SC7 the distance D from the stereo camera 2 to the subject is determined for each pair of the reference point and the corresponding point obtained by the P detection process in step SB6 by the distance measuring unit 423C according to the determination result in step SC62. Derived. For example, the distance D is not derived for each pair of reference points and corresponding points whose reliability is less than the threshold, and the distance D is derived for each pair of reference points and corresponding points whose reliability is equal to or greater than the threshold.
- the reliability is determined for each set of the reference point and the corresponding point obtained by the P detection process. For this reason, it is possible to take measures for corresponding points with low reliability.
- a reference point and a corresponding point corresponding to this reference point are indirectly obtained based on the detection result of I detection processing and motion information. It was.
- the corresponding points obtained indirectly are set as temporary corresponding points (also referred to as temporary corresponding points), and the temporary corresponding points are set as processing target points.
- Corresponding point detection processing for a set of P pictures is performed. By such a P detection process having two steps, a decrease in the detection accuracy of corresponding points is suppressed, and the amount of calculation related to the detection of corresponding points is reduced.
- differences from the information processing system 1D according to the third embodiment in the information processing system 1D according to the fifth embodiment will be described.
- FIG. 34 is a diagram illustrating a functional configuration of the information processing system 1D.
- the information processing system 1D is based on the information processing system 1B according to the third embodiment, and the calculation unit 42B is changed to a calculation unit 42D in which the search unit 422B is replaced with the search unit 422D.
- the search unit 422D includes an I detection unit 4221D, a temporary detection unit 4222D, and a P detection unit 4223D.
- the I detection unit 4221D performs the same process as the I detection process according to the third embodiment.
- the provisional detection unit 4222D as the first detection unit performs the same processing as the P detection processing according to the third embodiment on the other corresponding point corresponding to the reference point included in one P picture for the set of P pictures. From the P picture. That is, the detection result of the I detection process for a set of P pictures, the set of I pictures or the set of I pictures and D pictures, and the motion information included in the set of P pictures Based on the above, a temporary corresponding point corresponding to each reference point on one P picture is obtained from the other P picture.
- the P detection unit 4223D as the second detection unit targets a set of P pictures and includes a region including a reference point included in one P picture (also referred to as a reference region) and the other P picture.
- the corresponding point corresponding to the reference point of one P picture is detected from the other P picture by comparison with a region including the temporary corresponding point (also referred to as a comparison region).
- a reference region W1 including a reference point (eg, reference point Pa2) as a center is set in one P picture (eg, P picture Ma2), and the other A comparison area W2 including a temporary corresponding point (for example, the temporary corresponding point Pb2) as a center is set in the P picture (for example, the P picture Mb2).
- the distribution of correlation values between the reference area W1 and the comparison area W2 by the same method as the method for obtaining the distribution of correlation values between the reference area Wa and the comparison area Wb in the P detection process according to the first embodiment.
- a point corresponding to the maximum correlation value in the distribution is detected as a corresponding point.
- step SB65 (FIG. 24) of the operation flow according to the third embodiment is replaced with step SD65 in which the operation flow shown in FIG. 36 is executed, and accordingly step SB6 is performed.
- step SD65 (FIG. 22) is replaced with step SD6 in which the processing content is changed.
- This operation flow is realized by the control unit 406 reading and executing the program PGD in the storage unit 404.
- step SD65 different from the operation flow according to the third embodiment will be described.
- step SD65 the processes of steps SD651 and SD652 shown in FIG. 36 are sequentially performed.
- step SD651 the provisional detection unit 4222D detects a corresponding point corresponding to the reference point included in one P picture from the other P picture for the set of P pictures.
- step SD652 the P detection unit 4223D causes a reference region including the reference point included in one P picture as a center, and the provisional corresponding point included in the other P picture detected in step SD651.
- the corresponding point corresponding to the reference point of one P picture is detected from the other P picture.
- the weighting coefficient set for each frequency is multiplied by the value after the synthesis process T4.
- a mode in which processing for limiting a component of a predetermined frequency also referred to as band limitation processing
- this weighting coefficient a weighting coefficient distributed in a rectangular shape that completely cuts a high frequency component as shown in FIG. 37 may be adopted, or a Gaussian shape as shown in FIG. A distributed weighting factor may be employed.
- the frequency neglected in the band limiting process may be set in accordance with the frequency information deleted by the moving image compression executed by the compression units 22a and 22b. That is, the frequency neglected in the band limiting process may be set according to the compression rate in moving image compression.
- the band control process As described above, it is possible to reduce the amount of calculation related to the frequency of the predetermined band in the I detection process. For this reason, the amount of calculation related to detection of corresponding points is reduced by reducing unnecessary calculations. Then, by reducing the influence of so-called aliasing and the like, the detection accuracy of corresponding points in the I detection process can be improved.
- the P detection process the same operation and effect as the I detection process can be obtained if the same band limiting process as the I detection process is performed.
- 39 and 40 are diagrams illustrating examples of how the reference area Wa and the comparison area Wb are set according to the block. 39 and 40, each square area surrounded by a broken line corresponds to a block in each frame. As shown in FIG. 39, if the reference area Wa and the comparison area Wb are set so as to match the area corresponding to one block, the reference area Wa and the comparison area Wb do not straddle the boundary line between the blocks. . For this reason, the fall of the detection accuracy of a corresponding point is suppressed notably. However, since the center point of the reference area Wa is set as a reference point, the reference points are set in order at discrete positions in each frame.
- the reference area Wa and the comparison area Wb are set so as to match an area corresponding to a block group composed of a plurality of (for example, four) blocks. For example, the boundary line between the blocks included in the reference area Wa and the comparison area Wb is reduced. For this reason, the fall of the detection accuracy of a corresponding point is suppressed.
- the reference areas Wa S1 since the area corresponding to one block is small, as shown in FIG. 40, the reference areas Wa S1 .
- a mode in which Wa S3 and comparison regions Wb S1 to Wb S3 are set is preferable. That is, the outer edge of the block may be matched with the outer edges of the reference area Wa and the comparison area Wb.
- the reference points are set discretely, and the processing target point is slightly off from the center. There is a tendency that comparison areas Wb, Wb S1 to Wb S3 are included.
- a temporary corresponding point is obtained indirectly based on motion information, and this temporary corresponding point is set as a processing target point, and corresponding points for a set of P pictures.
- the present invention is not limited to this.
- a P picture set is expanded into a D picture set, and the temporary corresponding point is set as a reference for the D picture set.
- Corresponding points may be detected in the same manner.
- a complicated process of generating a multi-step resolution frame and detecting a corresponding point is not required for a set of D pictures, and the corresponding point detection accuracy is maintained. That is, by the P detection process having the two-stage process, the calculation amount related to the detection of the corresponding point is reduced while the decrease in the detection accuracy of the corresponding point is suppressed.
- the difference between the information processing system 1E according to the third modification and the information processing system 1D according to the fifth embodiment will be described.
- FIG. 41 is a diagram illustrating a functional configuration of the information processing system 1E.
- the information processing system 1E is based on the information processing system 1D according to the fifth embodiment, and the calculation unit 42D is changed to a calculation unit 42E in which the search unit 422D is replaced with the search unit 422E and an expansion unit 426E is added. It has been done.
- the search unit 422E is obtained by replacing the P detection unit 4223D with the I detection unit 4223E based on the search unit 422D according to the fifth embodiment.
- the decompression unit 426E decompresses one P picture of the set of P pictures to generate a decompressed frame (that is, D picture), and decompresses the other P picture to decompress the decompressed frame (that is, D picture) ) Is generated.
- the I detection unit 4223E as the second detection unit targets a set of D pictures, and includes an area including a reference point included in one D picture (also referred to as a reference area) and the other D picture. Corresponding points corresponding to the reference point of one D picture are detected from the other D picture by comparison with a region (also referred to as a comparison region) that includes temporary corresponding points (that is, processing reference points). Specifically, the correlation value between the reference region and the comparison region is calculated by a method similar to the method in which the correlation value distribution between the reference region Wa and the comparison region Wb is obtained in the I detection process according to the first embodiment. It is conceivable that a distribution is obtained and a point corresponding to the maximum correlation value in the distribution is detected as a corresponding point.
- step SD65 (FIG. 24) is replaced with step SE65 in which the operation flow shown in FIG. 42 is executed.
- Step SD6 (FIG. 22) is replaced with Step SE6 in which the processing content is changed.
- This operation flow is realized by the control unit 406 reading and executing the program PGE in the storage unit 404.
- step SE65 different from the operation flow according to the fifth embodiment will be described.
- step SE65 the processing of steps SE651 to SE653 shown in FIG. 42 is performed in order.
- step SE651 similarly to step SD651 according to the fifth embodiment, the provisional detection unit 4222D targets the set of P pictures, and the corresponding point corresponding to the reference point included in one P picture is the other. Detected from P picture.
- step SE652 the expansion unit 426E expands one P picture of a set of P pictures to generate one D picture, and expands the other P picture to generate the other D picture. The Thereby, a set of D pictures is generated.
- step SE653 the I detection unit 4223E targets the set of D pictures generated in step SE652, the reference region including the reference point on one D picture, and the other detected in step SE651. Corresponding points corresponding to the reference point of one D picture are detected from the other D picture by comparison with a comparison region including a temporary corresponding point (that is, a processing reference point) on the D picture.
- a set of an I picture and a P picture is converted into a D picture by expanding the P picture, and then an I detection is performed on the set of the I picture and the D picture.
- the I detection may be performed on the set of P pictures after the I picture is converted into a P picture by moving image compression. That is, if the data format of the frame is converted by the decompression unit 421 so that the pair of the I picture and the P picture is matched with the data format of one of the I picture and the P picture. good. According to this, it is possible to detect corresponding points even for sets of frames having different forms.
- 43 and 44 are diagrams illustrating an aspect in which each frame of the second compressed moving image Ma is converted as necessary so as to match the data format of each frame of the first compressed moving image Ma.
- the first compressed moving image Ma is left as it is. It is treated as one compressed moving image Maa, and for the second compressed moving image Mb, the data format of the first, third, and ninth frames Mb1, Mb3, and Mb9 is converted to generate the second compressed moving image Mbb.
- the P picture is expanded and converted into D pictures Md1 and Md9.
- the I picture is compressed to generate a frame (also referred to as E picture) Me3 indicated by the motion information in the same manner as the P picture.
- the P detection process may be performed for the set of the P picture Ma3 and the E picture Me3.
- the previous P picture Mb2 is expanded into a D picture, and each picture is indicated by motion information obtained from the comparison between the D picture and the I picture Mb3. It suffices if Me3 is generated.
- the processing according to this modified example has a small proportion of I pictures in a large number of frames constituting each compressed video, It is preferably employed when the frequency of generation is low.
- the distance D is not obtained for each pair of the reference point and the corresponding point determined to be low in reliability by the reliability determination unit 425.
- the present invention is not limited to this.
- other processing may be performed according to the determination result by the reliability determination unit 425.
- the distance measurement unit 423C adds a flag indicating that the reliability is low with respect to the distance D obtained for each pair of the reference point and the corresponding point determined to be low in reliability. And the like.
- the search unit 422B changes the set of reference points and corresponding points into a set of reference points and processing target points.
- the corresponding point detection process may be performed.
- a reference area Wa including the reference point as a center in one P picture and a process target point as a center are included in the other P picture.
- one and the other P picture are expanded into a set of D pictures, and this set of D pictures is targeted.
- Corresponding points may be detected by performing I detection processing.
- the P picture is adopted as the prediction frame, but the present invention is not limited to this.
- any one of a P picture and a B picture may be adopted as the prediction frame.
- the distance D from the stereo camera 2 to the subject is obtained, but the present invention is not limited to this.
- an image of the subject that is predicted to be obtained if captured from a viewpoint different from the viewpoint of the stereo camera 2 is generated in a pseudo manner May be.
- the I detection process uses the POC method, but is not limited to this.
- a so-called SAD method may be used.
- the SAD method is adopted for the I detection process, the corresponding point is detected while the temporary corresponding point is used as a reference in the I detection process for a set of D pictures.
- each combination of a reference point and a corresponding point is detected for a stereo moving image.
- Such a detection result may be stored in the storage unit 404 in a multi-view video coding (MVC) format.
- MVC multi-view video coding
- an image related to one viewpoint is set as a base image, and an image related to another viewpoint is expressed by disparity information with respect to the base image.
- Each image is subjected to compression by motion compensation.
- the corresponding point corresponding to the reference point is detected between two compressed moving images such as the first and second compressed moving images.
- a corresponding point corresponding to the reference point may be detected among three or more compressed moving images. That is, a corresponding point corresponding to the reference point may be detected among a plurality of compressed moving images.
- the first and second compression units 22a and 22b exist separately.
- the present invention is not limited to this, and a single compression unit may be used.
- the I detection process is performed on the set of I pictures and the set of I pictures and D pictures.
- the present invention is not limited to this. For example, if a P picture occupies a large proportion in each compressed video, one or more sets of P pictures are expanded into D picture sets, and an I detection process is performed on each D picture set. Also good.
- Information processing system 4 Information processing device 21a, 21b Imaging unit 22a, 22b Compression unit 23a, 23b Transmission unit 41a, 41b Reception unit 42, 42A to 42E Calculation unit 404 Storage unit 406 Control unit 421, 426E Expansion unit 422, 422A, 422B, 422D, 422E Search unit 424 Evaluation value calculation unit 425 Reliability determination unit 4221D, 4223E I detection unit 4222D Temporary detection unit 4223D P detection unit PG, PGA to PGE program
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Measurement Of Optical Distance (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
Description
<(1-1)画像処理システムの概要>
図1は、本発明の第1実施形態に係る情報処理システム1の概略構成を示す図である。
図2は、情報処理システム1の機能的な構成を示す図である。
第1カメラ2aは、撮像部21aと圧縮部22aと送信部23aとを備え、第2カメラ2bは、撮像部21bと圧縮部22bと送信部23bとを備える。ここでは、撮像部21aと撮像部21bとが同様な構成を有し、圧縮部22aと圧縮部22bとが同様な構成を有し、送信部23aと送信部23bとが同様な構成を有しているため、説明の重複を避ける目的で、撮像部21aと圧縮部22aと送信部23aとについて主に説明する。
図2で示されるように、画像処理装置としての情報処理装置4は、受信部41a,41bと演算部42とを備えている。
図5は、第1および第2圧縮動画像Maa,Mbbに含まれている各フレームの形態を例示する模式図である。各フレームでは、多数の画素がマトリックス状に配列されている。具体的には、縦方向(Y方向)に第1所定数(ここでは480個)の画素が配列され、横方向(X方向)に第2所定数(ここでは640個)の画素が配列されている。各フレームでは左上の位置が原点とされ、各フレームを構成する各画素の横方向における位置がX座標で示され、縦方向における位置がY座標で示されている。つまり、各フレームでは、各画素の位置が座標(X,Y)で示され、例えば、右方向(X方向)に1画素ずれるとX座標の値が1つ増加し、下方向(Y方向)に1画素ずれるとY座標の値が1つ増加する。
図9で示されるように、一方フレームに対して、対応点の検出の基準となる点(基準点とも言う)Sp1が設定される。ここでは、一方フレームの左上端の画素が始点とされ、図9の矢印で示されるように、一方フレームに対して、1つの水平ラインに沿って基準点Sp1が左端から右端にかけて1画素ずつずらされながら順に設定され、1つの水平ラインに沿った基準点Sp1の設定が完了すると、1つ下側(+Y側)の水平ラインに沿って基準点Sp1が左端から右端にかけて1画素ずつずらされながら順に設定される動作が繰り返される。また、ここでは、1つの基準点Sp1に対応する対応点が、他方フレーム上で検出される度に、次の基準点Sp1が設定される。
各基準点Sp1に対応する対応点の検出では、次の工程(C1)~(C12)が行われる。
探索部422で実行されるPOC法を用いた対応点の検出方法には、I検出処理に係る検出方法と、P検出処理に係る検出方法とが含まれる。
図17は、I検出処理に係るPOC法を用いた対応点の検出方法を具体的に説明するための図である。ここでは、一方フレームと他方フレームとの間における対応点の検出方法を例に挙げて説明する。
P検出処理では、一組のPピクチャに含まれる動き情報が用いられて、基準点に対応する対応点が検出される。以下、具体的に説明する。
図22から図24は、情報処理システム1における動作フローを示すフローチャートである。本動作フローは、制御部406がプログラムPGを読み込んで実行することで実現される。なお、本動作フローは、例えば、操作部401からの指示に応じて開始されて、図22のステップS1に進む。
以上のように、第1実施形態に係る情報処理システム1では、Pピクチャどうしの組についてはPピクチャが伸長されずにP検出処理が行われる。このため、圧縮された動画像を対象とした対応点の検出に係る演算量が低減される。また、IピクチャとPピクチャとの組に含まれたPピクチャが伸長されることでDピクチャとされてI検出処理が行われる。このため、演算量の増大が抑制されつつ異なる形態のフレームの組についても対応点の検出が可能となる。
第1実施形態に係る情報処理システム1では、IピクチャとPピクチャとの組については、Pピクチャが伸長されてDピクチャとされた上で、I検出処理が行われた。これに対して、第2実施形態に係る情報処理システム1Aでは、IピクチャとPピクチャとの組が生じない動画圧縮が行われる。これにより、フレームの形態を整合させる演算が削減され、対応点の検出に係る演算量が低減される。以下、第2実施形態に係る情報処理システム1Aのうち、上記第1実施形態に係る情報処理システム1と異なる点について説明する。
図25は、情報処理システム1Aの機能的な構成を示す図である。
図27および図28は、情報処理システム1Aにおける動作フローを示すフローチャートである。本動作フローは、制御部406が記憶部404内のプログラムPGAを読み込んで実行することで実現される。なお、本動作フローは、例えば、操作部401からの指示に応じて開始されて、図27のステップS1に進む。
以上のように、第2実施形態に係る情報処理システム1Aでは、第1および第2動画像に対する動画圧縮の同期によって、Pピクチャの伸長が不要となる。このため、フレームの形態を整合させる演算が削減され、対応点の検出に係る演算量が低減される。
第1および第2実施形態に係る情報処理システム1,1Aでは、Pピクチャの組については、一方のPピクチャの動き情報と他方のPピクチャの動き情報との間における相関に基づいて対応点が検出された。これに対して、第3実施形態に係る情報処理システム1Bでは、I検出処理の結果とPピクチャが有する動き情報とに基づき、Pピクチャの組について対応点が検出される。つまり、P検出処理の内容が異なる。これにより、P検出処理に必要な演算量が顕著に削減され、Pピクチャに係る対応点の検出に要する演算量が大幅に低減される。以下、第3実施形態に係る情報処理システム1Bのうち、第1実施形態に係る情報処理システム1と異なる点について説明する。
第3実施形態に係る情報処理システム1Bでは、Pピクチャの組については、I検出処理の検出結果と動き情報とに基づいて、基準点とこの基準点に対応する対応点とが間接的に求められた。これに対して、第4実施形態に係る情報処理システム1Cでは、間接的に求められた基準点と対応点とについて、その対応関係の信頼性が判定される。これにより、信頼性が低い対応点について、対応を講ずることが可能となる。以下、第4実施形態に係る情報処理システム1Cのうち、第3実施形態に係る情報処理システム1Bと異なる点について説明する。
図31は、情報処理システム1Cの機能的な構成を示す図である。
図33は、情報処理システム1Cにおける動作フローを示すフローチャートである。本動作フローは、制御部406が記憶部404内のプログラムPGCを読み込んで実行することで実現される。本動作フローでは、まず、図22で示された上記第3実施形態に係るステップS1~S5,SB6と同様な処理が行われ、その後、ステップSC61,SC62,SC7の処理が順に行われる。
以上のように、第4実施形態に係る情報処理システム1Cでは、P検出処理で求められた基準点と対応点との各組について、信頼性が判定される。このため、信頼性が低い対応点について、対応を講ずることが可能となる。
第3実施形態に係る情報処理システム1Bでは、Pピクチャの組については、I検出処理の検出結果と動き情報とに基づき、基準点とこの基準点に対応する対応点とが間接的に求められた。これに対して、第5実施形態に係る情報処理システム1Dでは、間接的に求められた対応点が仮の対応点(仮対応点とも言う)とされ、この仮対応点を処理対象点として、Pピクチャの組を対象とした対応点の検出処理が行われる。このような2段階の処理を有するP検出処理により、対応点の検出精度の低下が抑制されつつ、対応点の検出に係る演算量が低減される。以下、第5実施形態に係る情報処理システム1Dのうち、第3実施形態に係る情報処理システム1Bと異なる点について説明する。
図34は、情報処理システム1Dの機能的な構成を示す図である。
情報処理システム1Dにおける動作フローは、上記第3実施形態に係る動作フローのステップSB65(図24)が、図36で示される動作フローが実行されるステップSD65に置換され、それに伴って、ステップSB6(図22)が、処理内容が変更されたステップSD6に置換されたものとなる。本動作フローは、制御部406が記憶部404内のプログラムPGDを読み込んで実行することで実現される。以下では、上記第3実施形態に係る動作フローと異なるステップSD65について説明する。
以上のように、第5実施形態に係る情報処理システム1Dでは、P検出処理において、I検出処理と動き情報とから求められる仮対応点が用いられることで、多段階の解像度のフレーム間における対応点の検出が行われずとも、対応点が容易に検出される。このため、対応点の検出精度の低下が抑制されつつ、対応点の検出に係る演算量が低減される。
なお、本発明は上述の実施の形態に限定されるものではなく、本発明の要旨を逸脱しない範囲において種々の変更、改良等が可能である。以下、第1~5変形例について順に説明した後に、その他の変形例について説明する。
圧縮部22a,22bで実行される動画圧縮では、上述したように、所定帯域(主に高周波帯域)の周波数に係る情報が削除される。つまり、各Iピクチャは、所定帯域の周波数に係る情報を削除する圧縮処理が施されたフレームである。このため、第1~5実施形態に係る演算部42,42A~42Dで実行されるI検出処理において、所定帯域を軽視したPOC法を用いた演算が行われても良い。ここで、所定帯域を軽視したPOC法を用いた演算では、所定帯域について、残余の帯域よりも周波数成分が低減されるような重み付けが行われる。
圧縮部22a,22bで実行される動画圧縮では、上述したように、DCTや量子化が所定サイズのブロック単位で行われる。このため、動画圧縮後の各フレームには、所謂ブロックノイズが存在する。そこで、ブロックノイズの影響による対応点の検出精度の低下が抑制される観点から言えば、I検出処理およびP検出処理において各フレームに対して設定される基準領域Waおよび比較領域Wbは、ブロックに合わせた領域として設定されることが好ましい。
上記第5実施形態に係る情報処理システム1Dでは、動き情報に基づいて間接的に仮対応点が求められ、この仮対応点が処理対象点とされて、Pピクチャの組を対象とした対応点の検出処理が行われたが、これに限られない。
上記第1~5実施形態では、IピクチャとPピクチャとの組については、Pピクチャが伸長されることでDピクチャとされた上で、IピクチャとDピクチャとの組を対象としたI検出処理が行われたが、これに限られない。例えば、IピクチャとPピクチャとの組について、Iピクチャが動画圧縮によってPピクチャに変換された上で、Pピクチャの組を対象としたP検出処理が行われても良い。すなわち、伸長部421によって、IピクチャとPピクチャとの組については、IピクチャおよびPピクチャのうちの何れか一方のピクチャのデータ形式に合わせられるように、フレームのデータ形式の変換が行われれば良い。これによれば、異なる形態のフレームの組についても対応点の検出が可能となる。
◎例えば、上記第4実施形態では、信頼性判定部425によって信頼性が低いものと判定された基準点と対応点との各組については距離Dが求められなかったが、これに限られない。例えば、信頼性判定部425による判定結果に応じて、その他の処理が行われても良い。その他の処理としては、例えば、測距部423Cが、信頼性が低いものと判定された基準点と対応点との各組について求めた距離Dに対して信頼度が低い旨を示すフラグを付する処理等が挙げられる。
4 情報処理装置
21a,21b 撮像部
22a,22b 圧縮部
23a,23b 送信部
41a,41b 受信部
42,42A~42E 演算部
404 記憶部
406 制御部
421,426E 伸長部
422,422A,422B,422D,422E 探索部
424 評価値算出部
425 信頼性判定部
4221D,4223E I検出部
4222D 仮検出部
4223D P検出部
PG,PGA~PGE プログラム
Claims (15)
- 基準フレームと、他のフレームを基準とした動き情報によって各画素が示されている予測フレームと、をそれぞれ有する第1および第2圧縮動画像を取得する取得部と、
前記第1圧縮動画像と前記第2圧縮動画像との間におけるフレームの各組を対象として、前記第2圧縮動画像に含まれている一のフレームから、前記第1圧縮動画像に含まれている一のフレームの各基準点に対応する対応点をそれぞれ検出する検出処理を行う演算部と、を備え、
前記演算部が、
一組の前記予測フレームを対象とした前記検出処理を、該一組の前記予測フレームを示す動き情報を用いて行うことを特徴とする画像処理システム。 - 請求項1に記載の画像処理システムであって、
前記演算部が、
一組の前記予測フレームを対象とした前記検出処理を、一組の前記基準フレームを対象とした前記検出処理による検出結果と、該一組の前記予測フレームを示す動き情報とに基づいて行うことを特徴とする画像処理システム。 - 請求項2に記載の画像処理システムであって、
一組の前記予測フレームに係る前記検出処理において一方の予測フレームに含まれている一の基準点に対して他方の予測フレームから検出される一の対応点について、該一の基準点との類似度に係る評価値を算出する算出部と、
前記評価値と予め設定されている閾値との比較によって前記一の対応点に係る信頼性を判定する判定部と、
を更に備えることを特徴とする画像処理システム。 - 請求項1に記載の画像処理システムであって、
前記演算部が、
一組の前記予測フレームについて、一組の前記基準フレームを対象とした前記検出処理による検出結果と、該一組の前記予測フレームを示す動き情報とに基づいて、一方の予測フレームに含まれている一の基準点に対応する一の仮対応点を他方の予測フレームから求める第1検出部と、
前記一方の予測フレームにおいて前記一の基準点を包含する領域と、前記他方の予測フレームにおいて前記一の仮対応点を包含する領域との比較によって、前記他方の予測フレームから前記一の基準点に対応する一の対応点を検出する第2検出部と、
を有することを特徴とする画像処理システム。 - 請求項1に記載の画像処理システムであって、
前記演算部が、
一組の前記予測フレームについて、一組の前記基準フレームを対象とした前記検出処理による検出結果と、該一組の前記予測フレームを示す動き情報とに基づいて、一方の予測フレームに含まれている一の基準点に対応する一の仮対応点を他方の予測フレームから求める第1検出部と、
前記一方の予測フレームを伸長して一方の伸長後フレームを生成するとともに、前記他方の予測フレームを伸長して他方の伸長後フレームを生成する伸長部と、
前記他方の伸長後フレームにおいて前記一の仮対応点を包含する領域を基準として、前記他方の伸長後フレームから前記一の基準点に対応する一の対応点を検出する第2検出部と、
を有することを特徴とする画像処理システム。 - 請求項1から請求項5の何れか1つの請求項に記載の画像処理システムであって、
各前記基準フレームが、
所定帯域の周波数に係る情報を削除する圧縮処理が施されたフレームであり、
前記演算部が、
前記所定帯域について残余の帯域よりも周波数成分を低減する重み付けを行いつつ位相限定相関法を用いた演算を行うことで、一組のフレームを対象とした前記検出処理を行うことを特徴とする画像処理システム。 - 請求項1から請求項5の何れか1つの請求項に記載の画像処理システムであって、
各前記基準フレームが、
所定サイズのブロック単位で圧縮処理が施されたフレームであり、
前記演算部が、
前記第1圧縮動画像の第1フレームに対して一の基準点を包含する第1領域を前記ブロックに合わせて設定し、前記第2圧縮動画像の第2フレームに対して第2領域を前記ブロックに合わせて設定し、前記第1領域と前記第2領域とを対象とした位相限定相関法を用いた演算を行うことで、前記第1フレームと前記第2フレームとの組を対象とした前記検出処理を行うことを特徴とする画像処理システム。 - 請求項1から請求項7の何れか1つの請求項に記載の画像処理システムであって、
前記演算部が、
前記基準フレームと前記予測フレームとの組については、前記基準フレームおよび前記予測フレームのうちの何れか一方のフレームの形式に合わせるようにフレームの変換を行った上で、前記検出処理を行うことを特徴とする画像処理システム。 - 請求項8に記載の画像処理システムであって、
前記演算部が、
前記基準フレームと前記予測フレームとの組については、前記予測フレームを伸長した上で、前記基準フレームと伸長後の前記予測フレームとを対象とした前記検出処理を行うことを特徴とする画像処理システム。 - 請求項1に記載の画像処理システムであって、
前記一組の予測フレームが、
第1予測フレームと第2予測フレームとを含み、
前記演算部が、
前記第1予測フレームにおいて一の基準点を包含する基準領域に係る前記動き情報と、前記第2予測フレームにおいて一の仮対応点を包含する比較領域に係る前記動き情報との比較によって、前記第2予測フレームから前記一の基準点に対応する一の対応点を検出することを特徴とする画像処理システム。 - 請求項10に記載の画像処理システムであって、
前記基準領域に係る前記動き情報が、
一方向に係る動き情報の2次元的な一方基準分布と前記一方向とは異なる他方向に係る動き情報の2次元的な他方基準分布とを有し、
前記比較領域に係る前記動き情報が、
前記一方向に係る動き情報の2次元的な一方比較分布と前記他方向に係る動き情報の2次元的な他方比較分布とを有し、
前記演算部が、
前記一方基準分布および前記他方基準分布を含むボクセル情報と、前記一方比較分布および前記他方比較分布を含むボクセル情報との比較によって、前記第2予測フレームから前記一の基準点に対応する前記一の対応点を検出することを特徴とする画像処理システム。 - 請求項1から請求項11の何れか1つの請求項に記載の画像処理システムであって、
被写体を対象とした第1の一連の撮像によって第1動画像を得る第1撮像部と、
前記第1の一連の撮像と同一のタイミングにおける前記被写体を対象とした第2の一連の撮像によって第2動画像を得る第2撮像部と、
前記第1動画像に圧縮処理を施すことで前記第1圧縮動画像を生成する第1圧縮部と、
前記第2動画像に圧縮処理を施すことで前記第2圧縮動画像を生成する第2圧縮部と、
を更に備えることを特徴とする画像処理システム。 - 請求項1から請求項7の何れか1つの請求項に記載の画像処理システムであって、
第1動画像に圧縮処理を施すことで前記第1圧縮動画像を生成する第1圧縮部と、
第2動画像に圧縮処理を施すことで前記第2圧縮動画像を生成する第2圧縮部と、
を更に備え、
前記第1および第2圧縮部が、
第1動画像と第2動画像との間における各組のフレームが、一組の前記基準フレームおよび一組の前記予測フレームのうちの何れか一方の組となるように動作することを特徴とすることを特徴とする画像処理システム。 - (a)基準フレームと、他のフレームを基準とした動き情報によって各画素が示される予測フレームと、をそれぞれ有する第1および第2圧縮動画像を取得する工程と、
(b)前記第1圧縮動画像と前記第2圧縮動画像との間におけるフレームの各組を対象として、前記第2圧縮動画像に含まれている一のフレームから、前記第1圧縮動画像に含まれている一のフレームの各基準点に対応する対応点をそれぞれ検出する検出処理を行う工程と、を備え、
前記(b)工程において、
一組の前記予測フレームを対象とした前記検出処理が、該一組の前記予測フレームを示す動き情報を用いて行われることを特徴とする画像処理方法。 - 画像処理システムに含まれる制御部において実行されることにより、前記画像処理システムを、請求項1から請求項13の何れか1つの請求項に記載の画像処理システムとして機能させるプログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012510667A JP5299567B2 (ja) | 2010-04-15 | 2011-04-13 | 画像処理システム、画像処理方法、およびプログラム |
EP11768882A EP2560369A1 (en) | 2010-04-15 | 2011-04-13 | Image processing system, image processing method, and program |
US13/639,919 US8923639B2 (en) | 2010-04-15 | 2011-04-13 | Image processing system, image processing method, and program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010-093958 | 2010-04-15 | ||
JP2010093958 | 2010-04-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2011129361A1 true WO2011129361A1 (ja) | 2011-10-20 |
Family
ID=44798732
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/059160 WO2011129361A1 (ja) | 2010-04-15 | 2011-04-13 | 画像処理システム、画像処理方法、およびプログラム |
Country Status (4)
Country | Link |
---|---|
US (1) | US8923639B2 (ja) |
EP (1) | EP2560369A1 (ja) |
JP (1) | JP5299567B2 (ja) |
WO (1) | WO2011129361A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5110235B1 (ja) * | 2011-06-27 | 2012-12-26 | コニカミノルタホールディングス株式会社 | 画像処理装置、画像処理方法、およびプログラム |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3429186B1 (en) * | 2016-03-30 | 2022-08-24 | Huawei Technologies Co., Ltd. | Image registration method and device for terminal |
CN106331730B (zh) * | 2016-08-22 | 2020-03-06 | 上海交通大学 | H.264视频同量化因子双重压缩检测方法 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH07325924A (ja) * | 1994-06-02 | 1995-12-12 | Canon Inc | 複眼撮像装置 |
JP2004301607A (ja) | 2003-03-31 | 2004-10-28 | Honda Motor Co Ltd | 移動物体検出装置、移動物体検出方法及び移動物体検出プログラム |
JP2006284372A (ja) * | 2005-03-31 | 2006-10-19 | Toshiba Corp | 画像処理装置及び画像処理方法 |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH06350952A (ja) * | 1993-06-08 | 1994-12-22 | Hitachi Ltd | 圧縮動画像記録方法及び記録装置 |
US5592225A (en) * | 1993-09-30 | 1997-01-07 | Matsushita Electric Industrial Co., Ltd. | Device and method for controlling coding |
US6118475A (en) | 1994-06-02 | 2000-09-12 | Canon Kabushiki Kaisha | Multi-eye image pickup apparatus, and method and apparatus for measuring or recognizing three-dimensional shape |
EP2099228B1 (en) * | 2001-09-14 | 2014-11-12 | NTT DoCoMo, Inc. | Coding method, decoding method, coding apparatus, decoding apparatus, image processing system, coding program, and decoding program |
US7274825B1 (en) * | 2003-03-31 | 2007-09-25 | Hewlett-Packard Development Company, L.P. | Image matching using pixel-depth reduction before image comparison |
JP2010028221A (ja) * | 2008-07-15 | 2010-02-04 | Sony Corp | 動きベクトル検出装置、動きベクトル検出方法、画像符号化装置及びプログラム |
-
2011
- 2011-04-13 US US13/639,919 patent/US8923639B2/en not_active Expired - Fee Related
- 2011-04-13 JP JP2012510667A patent/JP5299567B2/ja not_active Expired - Fee Related
- 2011-04-13 WO PCT/JP2011/059160 patent/WO2011129361A1/ja active Application Filing
- 2011-04-13 EP EP11768882A patent/EP2560369A1/en not_active Withdrawn
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH07325924A (ja) * | 1994-06-02 | 1995-12-12 | Canon Inc | 複眼撮像装置 |
JP2004301607A (ja) | 2003-03-31 | 2004-10-28 | Honda Motor Co Ltd | 移動物体検出装置、移動物体検出方法及び移動物体検出プログラム |
JP2006284372A (ja) * | 2005-03-31 | 2006-10-19 | Toshiba Corp | 画像処理装置及び画像処理方法 |
Non-Patent Citations (1)
Title |
---|
TAKANORI SENO: "3D Kanren no MPEG Hyojunka Doko", IMAGE LAB, vol. 18, no. 2, 1 February 2007 (2007-02-01), pages 14, XP008161817 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5110235B1 (ja) * | 2011-06-27 | 2012-12-26 | コニカミノルタホールディングス株式会社 | 画像処理装置、画像処理方法、およびプログラム |
WO2013001885A1 (ja) * | 2011-06-27 | 2013-01-03 | コニカミノルタホールディングス株式会社 | 画像処理装置、画像処理方法、およびプログラム |
US9042655B2 (en) | 2011-06-27 | 2015-05-26 | Konica Minolta, Inc. | Image processing apparatus, image processing method, and non-transitory computer readable recording medium |
Also Published As
Publication number | Publication date |
---|---|
EP2560369A1 (en) | 2013-02-20 |
JPWO2011129361A1 (ja) | 2013-07-18 |
JP5299567B2 (ja) | 2013-09-25 |
US20130039595A1 (en) | 2013-02-14 |
US8923639B2 (en) | 2014-12-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10349060B2 (en) | Encoding video frames using generated region of interest maps | |
EP2632160B1 (en) | Method and apparatus for image processing | |
KR101210625B1 (ko) | 빈공간 채움 방법 및 이를 수행하는 3차원 비디오 시스템 | |
US9269163B2 (en) | Method and apparatus for compressing or decompressing light field images | |
US8786711B2 (en) | Image quality evaluation system, and apparatus, method and program used for the evaluation system | |
JP6127202B2 (ja) | 信号処理方法および装置 | |
US20210377507A1 (en) | Method and apparatus for processing data for three-dimensional image | |
JP7105062B2 (ja) | 画像処理装置、コンテンツ処理装置、コンテンツ処理システム、および画像処理方法 | |
JPWO2013054499A1 (ja) | 画像処理装置、撮像装置および画像処理方法 | |
US20150172715A1 (en) | Picture encoding method, picture decoding method, picture encoding apparatus, picture decoding apparatus, picture encoding program, picture decoding program, and recording media | |
JP2011129998A (ja) | 画像符号化装置、画像復号化装置、画像符号化方法、及び、画像復号化方法 | |
WO2009098741A1 (ja) | 撮像装置、集積回路及び撮像方法 | |
EP3813024A1 (en) | Image processing device and image processing method | |
EP3007448A1 (en) | Disparity data encoding apparatus and method of controlling the same for | |
JPWO2019198501A1 (ja) | 画像処理装置、画像処理方法、プログラム、および画像伝送システム | |
JP2019114842A5 (ja) | ||
JP5299567B2 (ja) | 画像処理システム、画像処理方法、およびプログラム | |
US7729543B2 (en) | Imaging apparatus | |
US9076189B2 (en) | Coding apparatus and method for encoding image data | |
KR100951637B1 (ko) | 스테레오 비전 시스템 및 이의 영상 처리 방법 | |
JP2016163169A5 (ja) | ||
JP2017103564A (ja) | 制御装置、制御方法、およびプログラム | |
WO2015023318A2 (en) | Depth image compression and decompression utilizing depth and amplitude data | |
JP5769248B2 (ja) | ステレオマッチング処理装置、ステレオマッチング処理方法、及び、プログラム | |
JP2019207464A (ja) | 位相差検出器及び画像処理装置、並びにプログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11768882 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012510667 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13639919 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REEP | Request for entry into the european phase |
Ref document number: 2011768882 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011768882 Country of ref document: EP |