WO2017033853A1 - 情報処理装置および情報処理方法 - Google Patents

情報処理装置および情報処理方法 Download PDF

Info

Publication number
WO2017033853A1
WO2017033853A1 PCT/JP2016/074207 JP2016074207W WO2017033853A1 WO 2017033853 A1 WO2017033853 A1 WO 2017033853A1 JP 2016074207 W JP2016074207 W JP 2016074207W WO 2017033853 A1 WO2017033853 A1 WO 2017033853A1
Authority
WO
WIPO (PCT)
Prior art keywords
distance
image
position information
acquisition unit
search range
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2016/074207
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
稲田　徹悟
寛史 岡本
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Interactive Entertainment Inc
Original Assignee
Sony Interactive Entertainment Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Interactive Entertainment Inc filed Critical Sony Interactive Entertainment Inc
Priority to EP16839200.9A priority Critical patent/EP3343500B1/en
Priority to US15/740,639 priority patent/US10559089B2/en
Publication of WO2017033853A1 publication Critical patent/WO2017033853A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication
    • G01C3/08—Use of electric radiation detectors
    • G01C3/085—Use of electric radiation detectors with electronic parallax measurement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication
    • G01C3/08—Use of electric radiation detectors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Definitions

  • the present invention relates to a technique for recognizing the position and movement of an object using a captured image.
  • stereo image method as a general method for obtaining the position of an object in a three-dimensional space.
  • corresponding points are detected from stereo images obtained by simultaneously capturing the same space with two cameras separated by a known distance on the left and right, and the distance from the imaging surface of the object is determined based on the parallax based on the principle of triangulation. This is a calculation method.
  • the present invention has been made in view of such problems, and an object of the present invention is to provide a technique capable of acquiring position information of an object in a three-dimensional space with high accuracy and high speed.
  • An aspect of the present invention relates to an information processing apparatus.
  • the information processing apparatus includes an image acquisition unit that acquires data of a stereo image obtained by photographing the same space with left and right cameras having a known interval, and a search block that is set for the reference block set for one of the stereo images.
  • a position information generating unit that detects a corresponding point by performing block matching to detect a region having a high degree of similarity, generates position information including a distance from the camera of the object based on the parallax, and outputs the position information.
  • the position information generation unit determines an estimated distance range from the camera of the target object, performs block matching after limiting the search range with a position and a length corresponding to the estimated distance range, and performing block matching .
  • This information processing method includes a step of acquiring data of a stereo image obtained by photographing the same space with left and right cameras having a known interval from an imaging device, and a search range set on the other side with respect to a reference block set on one side of the stereo image Detecting a corresponding point by performing block matching for detecting a region having a high similarity in the image, and generating and outputting position information including the distance from the camera of the target object based on the parallax, and The step of generating is to determine an estimated distance range from the camera of the target object, and to perform block matching after setting a limited search range with a position and length corresponding to the estimated distance range .
  • information processing based on the position and movement of an object in a captured image can be realized at high speed and with high accuracy.
  • the first position information acquisition unit detects an object at a predetermined position by setting a detection surface
  • the second position information acquisition unit describes the flow of processing when performing detailed stereo matching FIG.
  • FIG. 1 shows a configuration example of an information processing system to which this embodiment can be applied.
  • the information processing system 2 includes an imaging device 12 equipped with two cameras that photograph a subject such as the user 1, an information processing device 10 that performs information processing according to a user's request based on the captured image, and an information processing device 10 A display device 16 that outputs image data obtained as a result of the processing is included.
  • the information processing apparatus 10 may be connectable to a network such as the Internet.
  • the information processing apparatus 10, the imaging apparatus 12, and the display apparatus 16 may be connected by a wired cable, or may be wirelessly connected by a wireless LAN (Local Area Network) or the like. Any two or all of the imaging device 12, the information processing device 10, and the display device 16 may be combined and integrally provided. Further, the imaging device 12 is not necessarily installed on the display device 16. Further, the number and type of subjects are not limited.
  • the imaging device 12 has a configuration of a stereo camera in which two digital video cameras each provided with an imaging element such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor) are arranged on the left and right sides at a known interval. Each of the two digital video cameras captures a subject existing in the same space at a predetermined frame rate from the left and right positions.
  • a pair of frames shot in this way is also referred to as a “stereo image”.
  • the information processing apparatus 10 detects the position of the subject in the three-dimensional space including the image plane and the depth direction from the camera. The detection result is used for subsequent processing using the position and movement of the subject as input information. For example, it is used to realize an AR (augmented reality) in which a virtual object that reacts to the movement of the hand or foot of the user 1 as a subject is drawn on a captured image. Alternatively, the movement of the user 1 may be tracked and reflected in a game image or converted into an information processing command input. As described above, the purpose of using the information related to the position of the subject obtained in the present embodiment is not particularly limited.
  • the display device 16 displays the result of the processing performed by the information processing device 10 as an image as necessary.
  • the display device 16 may be a television having a display for outputting an image and a speaker for outputting sound, such as a liquid crystal television, a plasma television, a PC display, or the like.
  • the content of the process finally executed by the information processing apparatus 10 and the image to be displayed are not particularly limited depending on the purpose of use, and hence the focus is on the subject position detection process performed by the information processing apparatus 10 thereafter. Will be explained.
  • FIG. 2 shows an internal circuit configuration of the information processing apparatus 10.
  • the information processing apparatus 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26.
  • the main memory 26 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing. These units are connected to each other via a bus 30.
  • An input / output interface 28 is further connected to the bus 30.
  • the input / output interface 28 includes a peripheral device interface such as USB or IEEE1394, a communication unit 32 including a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or a nonvolatile memory, an output of the display device 16 or a speaker.
  • An output unit 36 for outputting data to the device an input unit 38 for inputting data from an input device such as a keyboard, a mouse, an imaging device 12 and a microphone, and a recording medium drive for driving a removable recording medium such as a magnetic disk, an optical disk or a semiconductor memory
  • the unit 40 is connected.
  • the CPU 22 performs information processing based on various programs read from the operating system and the removable recording medium stored in the storage unit 34 and loaded into the main memory 26 or downloaded via the communication unit 32. Execute. In addition, it controls processing and signal transmission in various circuits inside the information processing apparatus 10.
  • the GPU 24 has a function of a geometry engine and a function of a rendering processor, performs a drawing process according to a drawing command from the CPU 22, and stores display image data in a frame buffer (not shown). Then, the display image data is converted into a video signal and output to the output unit 36 or the like.
  • FIG. 3 shows a configuration of functional blocks of the imaging device 12 and the information processing device 10.
  • Each functional block shown in FIG. 3 can be realized in terms of hardware by the configuration of the CPU, GPU, memory such as RAM, and various circuits shown in FIG. 2, and in terms of software, it can be changed from a recording medium to the memory.
  • the loaded program is realized by a program that exhibits various functions such as a data input function, a data holding function, an image analysis function, and a drawing function. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.
  • the imaging device 12 includes a first camera 13a and a second camera 13b. Each camera captures a subject at a predetermined frame rate from left and right positions having a known interval. A stereo image obtained by photographing is transmitted to the information processing apparatus 10 at any time by a general method according to a request from the information processing apparatus 10.
  • the information processing apparatus 10 includes an image acquisition unit 42 that acquires data such as a stereo image from the imaging device 12, an input information acquisition unit 44 that acquires an instruction input from a user, and position information that generates position information of a subject based on a captured image.
  • Generation unit 46, output information generation unit 50 that performs necessary processing based on the position of the subject and generates output information, stereo image data acquired from imaging device 12, intermediate data generated by position information generation unit 46, and the like are stored
  • An image data storage unit 48 is included.
  • the input information acquisition unit 44 receives an instruction input from a user other than the imaging device 12 such as a process start / end request, a menu selection, a user operation on a game being executed, and the like, and receives a process request signal corresponding thereto. Send to other functional blocks.
  • the input information acquisition unit 44 is realized by the cooperation of a general input device such as a button, a keyboard, a mouse, a trackball, and a touch panel, and the CPU 22 that interprets the operation content performed on the input device and generates a processing request signal. To do.
  • the image acquisition unit 42 acquires data such as a stereo image from the imaging device 12 in accordance with a request from the input information acquisition unit 44 and stores it in the image data storage unit 48.
  • the data to be acquired may vary depending on the content of information processing performed by the information processing apparatus 10. For example, only an image captured by the first camera 13a may be acquired at a frame rate at the time of capturing, and stereo images captured by the first camera 13a and the second camera 13b may be acquired at a lower rate, that is, a frequency. That is, the acquisition rate of the image captured by the first camera 13a and the image captured by the second camera 13b may be set independently.
  • the image acquisition unit 42 is realized by the cooperation of the input unit 38, the input / output interface 28, and the CPU 22 shown in FIG.
  • the position information generation unit 46 is realized by the CPU 22, the GPU 24, and the like shown in FIG. 2, and generates information on the position of the subject in the three-dimensional space based on the stereo image data stored in the image data storage unit 48.
  • the position information generation unit 46 includes a first position information acquisition unit 52 and a second position information acquisition unit 56.
  • the position information generation process is divided into two processes, ie, a first process performed by the first position information acquisition unit 52 and a second process performed by the second position information acquisition unit 56. Improve and speed up processing. At this time, there are a case where the first process is a pre-process of the second process and a case where the first process and the second process are performed in parallel.
  • the first position information acquisition unit 52 specifies the approximate position of the subject by a predetermined means, and the second position information acquisition unit 56 sets the approximate position. Based on the target, detailed position information is acquired.
  • the second position information acquisition unit 56 acquires position information by a stereo image method, but the means for the first position information acquisition unit 52 to specify an approximate position may be various.
  • the second position information acquisition unit 56 limits the search range of corresponding points in the stereo image based on the range in which the target of interest is estimated.
  • the first position information acquisition unit 52 and the second position information acquisition unit 56 acquire the position information while paying attention to the objects that they are responsible for.
  • both the first position information acquisition unit 52 and the second position information acquisition unit 56 acquire position information by the stereo image method. And both limit the search range of the corresponding point in a stereo image based on the range in which the presence of the target object is estimated.
  • the second position information acquisition unit 56 also integrates the position information generated by the first process and the position information generated by the second process, and performs a process of generating final position information. .
  • the output information generation unit 50 is realized by the CPU 22, the GPU 24, and the like shown in FIG. 2, and further performs drawing processing on the captured image read from the image data storage unit 48 based on the position information of the subject generated by the position information generation unit 46. Appropriate processing according to the purpose of use is performed. As described above, the processing performed here is not particularly limited, and may be switched as appropriate according to an instruction from the user received by the input information acquisition unit 44, a program to be executed, or the like. The image data obtained as a result of the processing is output and displayed on the display device 16. Or you may transmit to another apparatus via a network. The output information generation unit 50 may further generate audio data corresponding to the movement of the subject and output it from a speaker.
  • FIG. 4 is a diagram for explaining the relationship between the parallax in the stereo image and the position of the subject in the depth direction.
  • the first camera 13a and the second camera 13b are installed so as to have parallel optical axes separated by a distance L. For these cameras, it is assumed that the subject is located at the rightmost arrow separated by a distance Z in the depth direction.
  • the width ⁇ x in the real space represented by one pixel of the image captured by each camera is expressed in proportion to the distance Z as follows.
  • ⁇ x Z ⁇ w / W
  • W is the number of pixels in the lateral direction of the camera
  • w is the visual field range in the lateral direction of the real space when the distance Z is unit length, and is determined by the viewing angle.
  • C is a value determined by the camera and its setting, and can be regarded as a constant during operation.
  • FIG. 5 is a diagram for explaining a basic block matching technique.
  • a reference block 182 having a predetermined size is set in, for example, the right viewpoint image 180b among the left viewpoint image 180a and the right viewpoint image 180b constituting the stereo image.
  • the size of the reference block 182 is, for example, 4 ⁇ 4 pixels, 8 ⁇ 8 pixels, or the like. Since the parallax D is obtained in units of reference blocks, the resolution of the position information in the depth direction is 1/4 ⁇ 1/4 times the original image if it is 4 ⁇ 4 pixels, and 1/8 ⁇ if it is 8 ⁇ 8 pixels. 1/8 times.
  • a block 184 having a higher degree of similarity than the reference block 182 of the right viewpoint image 180b is specified.
  • a search range 186 having a predetermined length in the horizontal direction is set. Considering that the image in the left viewpoint image 180a is closer to the right than the image of the same subject in the right viewpoint image 180b due to the parallax, the search range 186 is on the right side with the horizontal position x1 of the reference block 182 as the starting point.
  • the block frame having the same size as the reference block 182 is shifted in the horizontal direction within the search range 186, and the similarity between the area surrounded by the block frame (hereinafter referred to as “target block”) and the reference block 182 is calculated. I will do it.
  • the block frame is shifted by one pixel or by a predetermined plurality of pixels.
  • the target block when the highest similarity is obtained is the block 184 corresponding to the reference block 182.
  • the difference between the horizontal position x1 of the reference block 182 in the right viewpoint image 180b and the horizontal position x2 of the corresponding block 184 in the left viewpoint image 180a is the parallax D.
  • the process of obtaining the parallax D in this manner and obtaining the distance Z in the depth direction by the above relational expression is repeated by using each area formed by mesh-dividing the image plane of the right viewpoint image 180b as a reference block, thereby obtaining the right viewpoint image 180b.
  • a distance Z is associated with the position coordinates of each reference block.
  • SSD Sud-of-Squared-Difference
  • SAD Sud-of-Absolute-Difference
  • NCC Normalized-Cross-Correlation
  • ZNCC Zero-mean-Normalized-Cross-Correlation
  • the higher the resolution of the stereo image the more detailed and highly accurate the change in the similarity to the position of the target block is obtained.
  • the acquisition accuracy of the direction distance Z is improved.
  • the matching processing load for one reference block increases or the number of reference blocks increases, and the time required to determine the position of the subject in the entire area of the image also increases.
  • the search range 186 is limited to an appropriate position as described above, and highly accurate position information is generated with a small processing load. Next, the influence on the position information when the search range is limited will be described.
  • FIG. 6 shows a depth image of the position information acquisition result when the set position of the limited search range is changed.
  • the depth image is an image representing the distance Z in the depth direction of the subject obtained by the stereo image method as a luminance distribution on the image plane. In the figure, the greater the distance Z, the lower the luminance.
  • the reference block 182 set in the right viewpoint image 180b and the corresponding left block image 180a are set in the same format as shown in FIG.
  • the search range 186 is illustrated. This example is based on a photographed image centered on a person in the room, and the ceiling and surrounding objects are included in the field of view.
  • the depth image (a) when the start point xa of the search range 186 is set to the same position x1 as the reference block, the depth image (b) is several tens of pixels from the start point xb of the search range 186 to the position x1 of the reference block. It is a depth image when it is separated by minutes. It is assumed that the length of the search range is the same for both.
  • the depth image (a) has more defects in the data than the depth image (b).
  • the depth image (a) accurately indicates that the brightness is smaller than the face area 190a, that is, the depth is behind the face.
  • the depth image (b) is erroneously recognized as having the same luminance as the face area 190b, that is, at the same position in the depth direction.
  • the parallax D is inversely proportional to the distance Z in the depth direction, the parallax D increases as the object is closer to the front, and the corresponding points tend to move away from the reference block 182.
  • the search range 186 when the search range 186 is brought close to the reference block 182 as in the depth image (a), accurate position information can be obtained for the subject in the back. On the other hand, as the search range 186 is moved away from the reference block 182 as in the image (b), more accurate information can be obtained with respect to the nearer subject.
  • the similarity is good within the limited search range even though there is a true corresponding point where the highest similarity can be obtained.
  • another line or pattern may be misrecognized as a corresponding portion, and an incorrect distance value may be calculated.
  • the first position information acquisition unit 52 separately acquires the approximate position of the subject, and based on that, the second position information acquisition unit 56 sets an appropriate search range. Perform block matching. By setting the search range so that the true corresponding portion is included, accurate position information can be acquired even if the search range itself is short.
  • the process performed by the second position information acquisition unit 56 is (1) uses a higher resolution image than the process performed by the first position information acquisition unit 52, and (2) uses an advanced algorithm for calculating similarity. Use (3) Implement at a high frequency (rate).
  • the process performed by the first position information acquisition unit 52 can be simplified by making the second position information acquisition unit 56 a high-quality process.
  • FIG. 7 shows the flow of processing when the first position information acquisition unit 52 performs simple stereo matching and face detection processing, and the second position information acquisition unit 56 performs detailed stereo matching.
  • processing for detecting a corresponding portion on a stereo image by block matching as described above and obtaining a distance in the depth direction based on the parallax is also referred to as stereo matching.
  • the user who is in front of the imaging device 12 in the room is a main target for acquiring position information, but this is not intended to limit the target object or the shooting environment.
  • the first position information acquisition unit 52 performs simple stereo matching using them (S10).
  • Simple stereo matching is basically a process of generating a depth image by performing stereo matching in the same procedure as described above.
  • block matching is performed with a stereo image having a lower resolution than the second position information acquisition unit 56 by generating a low-resolution image by reducing the stereo images 194 a and 194 b acquired from the imaging device 12.
  • the block size during block matching may be increased.
  • the first position information acquisition unit 52 may use the low-resolution stereo image generated by the imaging device 12 in this way.
  • the first position information acquisition unit 52 may use a similarity calculation method that is simpler than the method used by the second position information acquisition unit 56 such as SAD.
  • SAD is a simpler calculation method than other calculation methods in which the sum of the differences between the values of corresponding pixels in the reference block and the target block is used as the similarity.
  • the first position information acquisition unit 52 may perform stereo matching at a lower frequency than the second position information acquisition unit 56.
  • the 1st position information acquisition part 52 does not limit the search range at the time of block matching. That is, a range wider than the second position information acquisition unit 56 from the reference block to the end of the image in the direction in which parallax occurs is set as the search range.
  • a simple depth image 196 representing the distance in the depth direction for all subjects in the field of view regardless of the position is obtained.
  • the first position information acquisition unit 52 performs face detection processing on one of the stereo images 194a and 194b (S12).
  • the face detection processing performed here may use a general technique such as template matching for comparing a captured image with a template image representing an average face image or a registered face image.
  • the frequency of the simple stereo matching process may be reduced accordingly.
  • the image of the object can be detected by matching with a template image representing the shape and feature points of the object instead of detecting the face.
  • the first position information acquisition unit 52 acquires the approximate value of the distance in the depth direction of the user including the face by applying the face area 198 on the image acquired by the face detection process to the simple depth image 196.
  • the approximate value of the distance is, for example, the pixel value of the pixel located at the center of the face area 198, or the average value of the pixel values constituting the face area within the face area 198 that can be considered to be at the same position. .
  • the second position information acquisition unit 56 performs detailed stereo matching after limiting the search region based on the approximate value of the distance (S16).
  • stereo images 194a and 194b with higher resolution than the first position information acquisition unit 52 are used, or the block size at the time of block matching is reduced.
  • the calculation method of the similarity is made more advanced than the method used by the first position information acquisition unit 52 such as ZNCC, or the process is performed at a higher frequency than the first position information acquisition unit 52.
  • any one of the resolution of the stereo image, the complexity of the calculation method, and the frequency of processing may be varied, or two or more may be varied simultaneously. Good.
  • how to make a difference may be appropriately determined.
  • the second position information acquisition unit 56 generates a detailed and highly accurate depth image 200 for at least the user as the object, and outputs the depth image 200 to the output information generation unit 50.
  • the output information generation part 50 can generate
  • FIG. 8 is a diagram for explaining the search range set by the second position information acquisition unit 56.
  • an approximate value of the distance in the depth direction of the user acquired by the first position information acquisition unit 52 is set to Ze.
  • the second position information acquisition unit 56 determines an estimated distance range Ze ⁇ Zf ⁇ Z ⁇ Ze + ⁇ Zb including a predetermined range before and after the approximate value Ze in consideration of the error of the approximate value and the thickness of the body.
  • ⁇ Zf is a margin in the front direction on the axis of the distance Z
  • the search range 204 set in the left viewpoint image 194a with respect to the reference block 202 at the position x1 in the right viewpoint image 194b is a range from x1 + Dmin to x1 + Dmax as the movement amount of the target block.
  • the second position information acquisition unit 56 uses each region obtained by dividing the image plane of the right viewpoint image 194b as a reference block, and calculates the similarity while shifting the target block within the search range 204 set for each region. Then, the target block when the high similarity is obtained is detected as a block corresponding to the reference block.
  • the depth image 200 in which the distance in the depth direction is expressed in detail and with high accuracy is obtained for an object that exists at least in the range of Ze ⁇ Zf ⁇ Z ⁇ Ze + ⁇ Zb.
  • the second position information acquisition unit 56 may further limit the setting range of the reference block 202. That is, by setting the reference block 202 only for the image area of the object, the block matching process itself for surrounding objects is omitted. In this case, it is necessary to specify the range that the image of the object covers on the image.
  • the movable range such as the hand and the toe can be easily estimated from the position and size of the face region 198 based on the face region 198 detected in S12. Therefore, the maximum range covered by the user image is determined so as to cover it, and the reference block 202 is set within the maximum range.
  • the simple stereo matching in S10 among the simple depth images 196 obtained, pixels having pixel values within a predetermined range from the values of the pixels constituting the face area 198 and continuous with the face area are obtained.
  • the area may be estimated as a human body area, and the reference block 202 may be set limited to the area.
  • the maximum range covered by the user is specified based on the approximate value of the distance obtained by the first position information acquisition unit 52, and the reference block 202 is set limited to that range.
  • FIG. 9 shows another example of the flow of processing by the position information generation unit 46 when the first position information acquisition unit 52 performs simple stereo matching and histogram generation, and the second position information acquisition unit 56 performs detailed stereo matching. It is a figure for demonstrating the flow of a process.
  • symbol is attached
  • the first position information acquisition unit 52 after performing simple stereo matching (S10), the first position information acquisition unit 52 generates a histogram of distance values using the obtained simple depth image 196 (S18).
  • a histogram representing the frequency of appearance in the image of the distance Z in the depth direction represented by the pixel value of the simple depth image 196 is generated.
  • a certain number of pixel groups representing the surface of the object in the depth image 196 have similar pixel values.
  • the first position information acquisition unit 52 compares a threshold value Th set in advance with a histogram, and extracts a portion having an appearance frequency equal to or higher than the threshold value Th.
  • the threshold value Th is determined by, for example, obtaining the number of pixels appearing as a human image from an actual photographed image and taking into account errors. Since the size of the image also changes depending on the distance in the depth direction, the threshold Th may be changed depending on the position in the depth direction in the histogram.
  • the position or range of the peak selected by a predetermined rule is estimated as the user position or range.
  • the peak that is closest to the user, that is, the distance Z is selected is selected.
  • the selection rule is not limited to this, and may be changed as appropriate depending on the shooting environment, the number of users, and the like. For example, by acquiring a histogram based on an image obtained by photographing only the background before the start of the game, a peak due to the background can be excluded during operation. In this case, even when there are a plurality of users, it can be recognized that a plurality of peaks other than the excluded peaks represent the image of each user.
  • the second position information acquisition unit 56 performs detailed stereo matching after limiting the search range based on the estimated distance range including a predetermined range before and after, similarly to the case described in FIG. 7 (S16). . Also in this case, since the detailed and highly accurate depth image 200 can be generated for at least the user who is the object, the second position information acquisition unit 56 outputs the data to the output information generation unit 50.
  • FIG. 10 shows another example of the flow of processing performed by the position information generation unit 46, in which the first position information acquisition unit 52 acquires an approximate value of distance through dialogue with the user, and the second position information acquisition unit 56 performs detailed stereo. It is a figure for demonstrating the flow of a process in the case of performing matching.
  • symbol is attached
  • the first position information acquisition unit 52 prompts the user to stand at a predetermined position whose distance in the depth direction is known by an instruction such as “Please stand near 1 m from the camera”. Or it guides to move to the position naturally on the game. Then, when the user stands at the position, the distance in the depth direction is set as the “approximately distance value” in the above-described manner.
  • the user may be allowed to input the distance from the imaging device 12 while sitting. Since the required distance may be an approximate value, it may be selected from a plurality of candidate distances having a predetermined step size such as every 30 cm.
  • the first position information acquisition unit 52 requests the output information generation unit 50 to express a standing position instruction or a distance input instruction to the user by an image or sound. When the standing position is instructed, the time when the user moves to the position and stops is determined and notified to the second position information acquisition unit 56.
  • the first position information acquisition unit 52 acquires the input information from the input information acquisition unit 44 and notifies the second position information acquisition unit 56.
  • the second position information acquisition unit 56 includes a predetermined range before and after the approximate value of the specified or input distance using the stereo images 194a and 194b photographed at that time as in the previous embodiments. Detailed stereo matching is performed after limiting the search range based on the estimated distance range (S16). Also in this case, since the detailed and highly accurate depth image 200 can be generated for at least the user who is the object, the second position information acquisition unit 56 outputs the data to the output information generation unit 50.
  • the timing at which the first position information acquisition unit 52 acquires the approximate value of the distance is limited to when the game is started. Therefore, the second position information acquisition unit 56 once acquires the approximate value of the distance and performs the detailed stereo matching, and then uses the position information acquired by itself for the previous frame to determine the position information of the subsequent frame. May be obtained.
  • FIG. 11 shows another example of the flow of processing performed by the position information generation unit 46.
  • the first position information acquisition unit 52 detects an object at a predetermined position by setting a detection surface
  • the second position information acquisition unit 56 FIG. 6 is a diagram for explaining a flow of processing when detailed stereo matching is performed.
  • symbol is attached
  • a method for detecting the position of an object using a detection surface is disclosed in Japanese Patent Laid-Open No. 2013-242812.
  • the detection surface is a surface virtually set in real space.
  • An area obtained by projecting the detection surface onto the screen coordinates of the stereo image has a parallax corresponding to the distance from the imaging device 12 to the detection surface. If a person stands at a position that overlaps with the detection surface, the image of the person in the stereo image should appear at the same position in two regions obtained by projecting the detection surface onto the stereo image. Therefore, a region obtained by projecting the detection surface is cut out from one of the stereo images, and a corresponding region determined by taking the parallax into consideration is cut out from the other to match the two. When it is determined that there is an image at the same position in the two regions, it can be seen that the object exists at the position of the detection surface. Further explanation will be given later.
  • the first position information acquisition unit 52 virtually sets such a detection surface at a predetermined position in real space, and performs matching on a region formed by projecting the detection surface on a stereo image. As a result, when a predetermined number or more of pixels having a high matching evaluation value are detected, it is determined that the user is at the position of the detection surface.
  • the detection surface By setting the detection surface at a plurality of positions, for example, in parallel with the image pickup surface of the image pickup device 12 at predetermined intervals, and performing similar determination processing on each detection surface, an approximate value of the distance in the depth direction of the user as the object Can be determined.
  • the second position information acquisition unit 56 performs detailed stereo matching after limiting the search range based on the estimated distance range including a predetermined range before and after the approximate value, similarly to the mode described so far (S16). Also in this case, since the detailed and highly accurate depth image 200 can be generated for at least the user who is the object, the second position information acquisition unit 56 outputs the data to the output information generation unit 50.
  • FIG. 12 is a diagram for explaining a method of detecting an object at a predetermined position by setting a detection surface.
  • the upper part of the figure is a schematic diagram 72a as seen from above the photographing environment, and the lower part is a schematic diagram 72b as seen from the side.
  • a person 74 as a subject faces the first camera 13a and the second camera 13b.
  • the equiparallax surface is distributed as indicated by the dotted line.
  • the equal parallax plane is a plane having the same parallax at all points on the plane.
  • the detection surface 76 is defined on the equiparallax surface at the position to be detected.
  • the detection surface 76 is a virtual plane in which vertex coordinates are defined in a three-dimensional space defined by a camera coordinate system.
  • the contour shape of the detection surface is not particularly limited, and the inclination may be arbitrarily set.
  • FIG. 13 shows an example of a stereo image taken in the environment shown in FIG.
  • a region 82 is obtained by projecting the detection surface 76 shown in FIG. 12 onto the left viewpoint image 80a.
  • an area obtained by projecting a “detection plane” defined in a three-dimensional space onto an image plane is referred to as a “detection area”.
  • the process of projecting a model defined in a three-dimensional space onto a two-dimensional image plane can be realized as a general computer graphics process.
  • the upper left coordinate of the detection area 82 is assumed to be (x, y).
  • a region where the same region as the detection region 82 is translated leftward by the amount of parallax and the upper left coordinate is (xC / Z, y) is defined as a parallax correction region 84. That is, the parallax correction area 84 is obtained by moving the area at the same position as the detection area 82 in a direction in which the parallax in the stereo image is eliminated. Then, a feature point image, for example, an edge image, is extracted from the detection region 82 of the left viewpoint image 80a and the parallax correction region 84 of the right viewpoint image 80b, and matching processing is performed. And the matching image 86 showing the pixel from which the high evaluation value was obtained is produced
  • the entire contour of the left hand of the subject extracted as an edge is shown by a solid line and a dotted line, but only the solid line portion of the actual matching image remains.
  • it is an image. That is, information that a part of the fingertip and a part of the wrist are located on the detection surface shown in FIG. 12 is obtained.
  • the portion to be left as a matching image is determined by threshold value determination of the matching evaluation value. By adjusting this threshold value, the detection resolution for the position Z in the depth direction can be controlled.
  • the parallax changes in the vertical direction of the image plane, and accordingly, the parallax correction region is determined by changing the shift amount of the detection region in the vertical direction accordingly. Then, the subsequent processing is the same.
  • the first position information acquisition unit 52 determines that the target is at the position of the detection surface when the number of pixels represented by the matching image 86 as being high in the matching evaluation value is equal to or greater than a predetermined value. The position is notified to the second position information acquisition unit 56 as an approximate value of the distance.
  • the detection surface has a size for detecting the position of the hand.
  • the size of the detection surface is also changed depending on the size of the object such as the entire body of the user. Furthermore, as described above, if a plurality of detection surfaces are set so that detection can be performed on any of the detection surfaces, an approximate value of the distance can be obtained wherever the object is located.
  • the simple stereo matching performed by the first position information acquisition unit 52 is an outline of the distance necessary for the second position information acquisition unit 56 to appropriately set a limited search range. The purpose was to get the value.
  • the simple depth image obtained as a result may be included in the position information data output from the position information generation unit 46. Specifically, among the depth images acquired by the second position information acquisition unit 56, the region of the image of the object whose distance has been obtained in detail and the simple depth image acquired by the first position information acquisition unit 52 Of these, a region other than the image of the object is combined to obtain a final depth image.
  • a plurality of areas representing depth information with different levels of detail and accuracy are mixed.
  • depth information with higher accuracy can be obtained for the region other than the region of the object such as the user as compared with the case where the search range is limited.
  • the required position recognition accuracy differs between the interaction with the user and the interaction with surrounding objects. That is, since the human body such as the user's hand has a complicated shape and a complicated movement, detailed positional information is required to realize a realistic interaction.
  • FIG. 14 is a diagram for explaining the flow of processing by the position information generation unit 46 in a mode in which depth information having different levels of detail and accuracy is combined to generate one depth image.
  • symbol is attached
  • the first position information acquisition unit 52 performs simple stereo matching using the stereo images 194a and 194b (S10).
  • the first position information acquisition unit 52 performs face detection processing on either of the stereo images 194a and 194b (S12).
  • the second position information acquisition unit 56 performs detailed stereo matching after limiting the search area in the same manner as described with reference to FIG. 7 (S16).
  • the second position information acquisition unit 56 generates the depth image 200 representing the detailed and highly accurate position information about the user as the object, and the simple stereo matching in which the first position information acquisition unit 52 performs S10.
  • the simple depth image generated by the above is synthesized (S24).
  • the pixel value of the region of the image of the object in the simple depth image 196 is the pixel value of the region corresponding to the depth image 200 acquired in detail by the second position information acquisition unit 56 for the object.
  • the resolution image size
  • the former is enlarged and the latter Replace the pixel value after aligning the size.
  • the update frequency of the composite portion of the simple depth image 196 is the composite portion of the detailed depth image 200.
  • the update frequency may be lower.
  • the update frequency of the former may be about 1/10 of the latter.
  • the frequency of the processing of the first position information acquisition unit 52 may be lowered.
  • the first position information acquisition unit 52 detects the face area as in FIG. 7, but the approximate value of the distance may be detected by a histogram as described in FIG. Further, as described with reference to FIGS. 10 and 11, it may be combined with a mode in which an approximate value of distance is obtained by means other than stereo matching. In this case, the first position information acquisition unit 52 separately performs simple stereo matching to generate a simple depth image as a synthesis target.
  • FIG. 15 is a diagram for explaining a synthesis processing technique.
  • the depth image generated by the second position information acquisition unit 56 is data specialized for the distance range of the object, and objects in other positions are excluded from the distance calculation target in principle.
  • By limiting the search range as described above only the objects that are originally present in the distance range corresponding to the search range are detected by the high degree of similarity in the block matching. Similar objects such as lines and patterns that appear repeatedly may be detected erroneously.
  • the upper part (a) of FIG. 15 shows how the first position information acquisition unit 52 performs block matching without limiting the search range
  • the lower part (b) of FIG. 15 shows that the second position information acquisition unit 56 limits the search range. It shows how block matching is performed.
  • a search range 224 that reaches the left viewpoint image 220a to the right end of the image is set for the reference block 222 of the right viewpoint image 220b.
  • a limited search range 234 is set for the left viewpoint image 230a with respect to the reference block 232 of the right viewpoint image 230b.
  • the search range is limited as shown in (b)
  • the maximum similarity B within the range at another position is obtained even though there is a true corresponding portion outside the search range 234.
  • the target block 238 at that time is detected as a corresponding block.
  • a false parallax is acquired, and thus a false distance value may be calculated.
  • the detailed depth image generated by the second position information acquisition unit 56 may include incorrect position information.
  • the final depth image 210 in which the position information with less error is represented in the entire image is generated by accurately determining the replacement area as described above.
  • an area having a pixel value equivalent to (predetermined range) the pixel value in the face area and consisting of a group of pixels continuous from the face area is the image of the user who is the object. Is replaced with the pixel value of the depth image generated by the second position information acquisition unit 56.
  • a pixel group that continuously forms a region of a predetermined size When using a histogram as described in FIG. 9, for example, among pixel groups that form a peak when an approximate value of distance is obtained, a pixel group that continuously forms a region of a predetermined size. The pixel value of the depth image generated by the second position information acquisition unit 56 is replaced.
  • the similarity obtained when the first position information acquisition unit 52 generates a simple depth image is compared with the similarity obtained when the second position information acquisition unit 56 generates a detailed depth image. If the latter is higher, replacement of detailed depth images with data may be performed.
  • the first position information acquisition unit 52 associates the maximum similarity A obtained when the target block 228 is detected as a corresponding block with the position of the corresponding reference block 222 in the simple stereo matching process. A degree image is generated in advance.
  • the second position information acquisition unit 56 associates the maximum similarity B obtained when the target block 238 is detected as the corresponding block with the position of the corresponding reference block 232. A similarity image is generated.
  • the similarity images corresponding to the reference block at the same position in the similarity images of both are compared, and the data of the depth image having the higher similarity is obtained. The distance value is used.
  • the distance value acquired by the first position information acquisition unit 52 according to (a) is adopted.
  • the first position information acquisition unit 52 and the second position information acquisition unit 56 calculate the similarity by different methods, it is conceivable that the similarity scales are different and cannot be simply compared. If the relationship between the two values is known in advance, one of them is converted to the other scale accordingly and then compared.
  • FIGS. 15A and 15B when a part of the search ranges 224 and 234 overlaps, the similarity at the same position is compared to obtain a magnification, and the result is multiplied by one. You may arrange the scale with.
  • the first position information acquisition unit 52 sets a wide search range in a stereo image and performs stereo matching with coarse accuracy as one method for obtaining an approximate value of the distance of the object.
  • the first position information acquisition unit 52 also calculates a distance value for an object existing at a position different from the second position information acquisition unit 56 by limiting the search range. May be combined into a final depth image.
  • FIG. 16 is a diagram for explaining the flow of processing by the position information generation unit 46 in a mode in which the distance calculation target is shared in this way.
  • the first position information acquisition unit 52 obtains the distance with respect to the background such as furniture and walls behind the user, and the second position information acquisition unit 56 displays the foreground of the user or the like as in the previous examples. Find the distance to the subject.
  • the search range set by the first position information acquisition unit 52 during stereo matching and the search range set by the second position information acquisition unit 56 during stereo matching are: Set individually for each distance range.
  • the former sets the search range up to 63 pixels starting from the position of the reference block
  • the latter sets the search range up to 127 pixels starting from the 64th pixel.
  • the search range may be a fixed value in this way, and as in the previous embodiments, the first position information acquisition unit 52 first obtains an approximate value of the distance of an object such as a user and searches for both accordingly.
  • the range may be set adaptively.
  • the search range set by the first position information acquisition unit 52 is the first search range
  • the search range set by the second position information acquisition unit 56 is the second search range.
  • the first search range and the second search range set for one reference block may or may not have overlapping portions as in the above example.
  • the first position information acquisition unit 52 performs stereo matching by setting the first search range in the other image 194a, for example, corresponding to each reference block of the one image 194b among the stereo images (S30).
  • a depth image 280 representing an accurate distance with respect to an object in the background is obtained.
  • the second position information acquisition unit 56 performs stereo matching by setting the second search range in the other image 194a corresponding to each reference block of the one image 194b among the stereo images (S32).
  • a depth image 282 in which an accurate distance to the object in the foreground, that is, the user is represented is obtained.
  • the 2nd position information acquisition part 56 synthesize combines both depth images similarly to the aspect demonstrated in FIG. 14 (S34). Specifically, in the depth image 280 generated by the first position information acquisition unit 52, the foreground area is replaced with data of the depth image 282 generated by the second position information acquisition unit 56. As a result, a final depth image 284 in which an accurate distance is expressed as a pixel value for both the foreground and the background is generated.
  • FIG. 17 illustrates the state of block matching when the first position information acquisition unit 52 and the second position information acquisition unit 56 share the object for which the distance is acquired.
  • (A) in the upper part of the figure illustrates the search range set by the first position information acquisition unit 52 corresponding to the two reference blocks 242 and 262.
  • (b) in the lower row exemplifies a search range set by the second position information acquisition unit 56 corresponding to the same reference blocks 242 and 262 as (a).
  • the way of representing the figure is the same as in FIG. However, in the figure, the position of the reference block is indicated by a black rectangle in the same image as that for setting the search range.
  • the first position information acquisition unit 52 determines the background and the second position information acquisition unit 56 determines the distance of the foreground
  • the former search range is close to the reference block
  • the latter search range is The range is far from the reference block.
  • the first position information acquisition unit 52 sets a search range 244 close to the reference block 242 as in an image 240.
  • the second position information acquisition unit 56 sets a search range 254 at a position away from the reference block 242 by the predetermined number of pixels d as described above, as in the image 250.
  • the target block 248 when the maximum similarity A is obtained among the similarities 246 obtained by the first position information acquisition unit 52 for the search range 244 is detected as a corresponding block.
  • the target block 258 when the maximum similarity B is obtained among the similarities 256 obtained by the second position information acquisition unit 56 for the search range 254 is detected as a corresponding block.
  • the value calculated by the first position information acquisition unit 52 is adopted as the distance value with respect to the reference block 242 by comparing the maximum similarities A and B as described in FIG. .
  • the similarity 266 obtained by the first position information acquisition unit 52 with respect to the search range 264 does not include a block whose similarity protrudes to such an extent that it can be regarded as corresponding, so that no corresponding portion is detected.
  • the pixel value of the depth image is 0 or the like.
  • the target block 278 when the maximum similarity D is obtained among the similarities 276 obtained by the second position information acquisition unit 56 for the search range 274 is detected as a corresponding block.
  • the value calculated by the second position information acquisition unit 56 is employed as the distance value for the reference block 262 during the synthesis process.
  • the degree of detail and accuracy of both processes can be individually set in the same manner as described above. Can be set. That is, as described above, since the foreground is assumed to have a more complicated shape and more movement than the background, the second position information acquisition unit 56 for the foreground is compared with the first position information acquisition unit 52 for the background.
  • the first position information acquisition unit 52 uses at least one of the aspects such as using a high-resolution stereo image, adopting an advanced technique for calculating the similarity, and increasing the frequency of processing.
  • the processing is divided into a plurality of roles in the information processing apparatus that obtains the position of the subject in the three-dimensional space using the stereo image. Specifically, the entire image or the object for which the distance is to be obtained in detail is obtained, the approximate value of the distance in the depth direction is obtained, and then the block matching is performed by limiting to the search range corresponding to the approximate value. In addition, highly accurate position information is obtained. Thereby, while being able to simplify a process with respect to objects, such as a background other than a target object, exact positional information is obtained about a target object. As a result, it is possible to reduce both the processing load and improve the accuracy of position information necessary for the subsequent processing.
  • the depth in which areas with different levels of detail and accuracy are mixed Generate an image.
  • the position information corresponding to it can be represented by one depth image.
  • the object whose distance is calculated is assigned according to the approximate position of the object, and an appropriate search range is set for each of the objects, and processing is performed in parallel. Then, by combining the position information generated by each, a final depth image that covers the necessary distance range is generated. At this time, at least one of resolution, similarity calculation method, and processing frequency is made different depending on the characteristics of the target object, so that unnecessary processing is omitted as much as possible, and processing is concentrated on the required target. be able to. As a result, it is possible to appropriately output accurate position information necessary for subsequent processing with a small processing load and data amount.
  • Information processing system 10 Information processing device, 12 Imaging device, 16 Display device, 13a 1st camera, 13b 2nd camera, 22 CPU, 24 GPU, 26 Main memory, 42 Image acquisition unit, 44 Input information acquisition unit, 46 Position information generation unit, 48 image data storage unit, 50 output information generation unit, 52 first position information acquisition unit, 56 second position information acquisition unit.
  • the present invention can be used for a game machine, an image processing apparatus, an information processing apparatus, an object recognition apparatus, an image analysis apparatus, or a system including any one of them.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Electromagnetism (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Measurement Of Optical Distance (AREA)
  • Length Measuring Devices By Optical Means (AREA)
PCT/JP2016/074207 2015-08-26 2016-08-19 情報処理装置および情報処理方法 Ceased WO2017033853A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP16839200.9A EP3343500B1 (en) 2015-08-26 2016-08-19 Image processing device and image processing method
US15/740,639 US10559089B2 (en) 2015-08-26 2016-08-19 Information processing apparatus and information processing method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2015167238A JP6548518B2 (ja) 2015-08-26 2015-08-26 情報処理装置および情報処理方法
JP2015-167238 2015-08-26

Publications (1)

Publication Number Publication Date
WO2017033853A1 true WO2017033853A1 (ja) 2017-03-02

Family

ID=58100144

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2016/074207 Ceased WO2017033853A1 (ja) 2015-08-26 2016-08-19 情報処理装置および情報処理方法

Country Status (4)

Country Link
US (1) US10559089B2 (enExample)
EP (1) EP3343500B1 (enExample)
JP (1) JP6548518B2 (enExample)
WO (1) WO2017033853A1 (enExample)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019082797A1 (ja) * 2017-10-23 2019-05-02 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 再構成方法および再構成装置
JP2024020074A (ja) * 2022-08-01 2024-02-14 トヨタ自動車株式会社 情報処理方法及び情報処理装置

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017176487A1 (en) 2016-04-08 2017-10-12 Orbital Insight, Inc. Remote determination of quantity stored in containers in geographical region
JP6899673B2 (ja) * 2017-03-15 2021-07-07 日立Astemo株式会社 物体距離検出装置
CN108230383B (zh) * 2017-03-29 2021-03-23 北京市商汤科技开发有限公司 手部三维数据确定方法、装置及电子设备
JP2018206063A (ja) * 2017-06-05 2018-12-27 株式会社東海理化電機製作所 画像認識装置及び画像認識方法
KR20190013224A (ko) * 2017-08-01 2019-02-11 엘지전자 주식회사 이동 단말기
JP6857147B2 (ja) * 2018-03-15 2021-04-14 株式会社日立製作所 三次元画像処理装置、及び三次元画像処理方法
JP7098365B2 (ja) * 2018-03-15 2022-07-11 キヤノン株式会社 画像処理装置、画像処理方法およびプログラム
JP7034781B2 (ja) * 2018-03-16 2022-03-14 キヤノン株式会社 画像処理装置、画像処理方法、及びプログラム
CN108629809B (zh) * 2018-04-26 2020-08-28 浙江大学 一种精确高效的立体匹配方法
CN111699361B (zh) * 2018-09-07 2022-05-27 深圳配天智能技术研究院有限公司 一种测量距离的方法及装置
JP7078172B2 (ja) * 2019-03-14 2022-05-31 オムロン株式会社 画像処理装置及び３次元計測システム
EP3832601B1 (en) 2019-03-14 2025-09-03 OMRON Corporation Image processing device and three-dimensional measuring system
JP7334051B2 (ja) * 2019-04-04 2023-08-28 東芝デベロップメントエンジニアリング株式会社 画像処理装置
JP7293100B2 (ja) * 2019-12-17 2023-06-19 日立Astemo株式会社 カメラシステム
JP2021135725A (ja) * 2020-02-26 2021-09-13 キヤノン株式会社 画像処理装置及びその制御方法並びにプログラム
CN113573039A (zh) * 2020-04-29 2021-10-29 思特威(上海)电子科技股份有限公司 目标深度值获取方法及双目系统
CN111724421B (zh) * 2020-06-29 2024-01-09 深圳市慧鲤科技有限公司 图像处理方法及装置、电子设备及存储介质
JP7679224B2 (ja) * 2021-04-28 2025-05-19 キヤノン株式会社 変位計及び物品の製造方法
CN118511538A (zh) * 2022-01-14 2024-08-16 松下知识产权经营株式会社 图像处理装置以及图像处理方法
JP2023166832A (ja) * 2022-05-10 2023-11-22 日立Astemo株式会社 画像処理装置及び画像処理方法
JP2025125773A (ja) 2024-02-16 2025-08-28 キヤノン株式会社 情報処理装置、情報処理装置の処理方法及びプログラム
CN118642101B (zh) * 2024-05-14 2025-12-19 西安电子科技大学 雷达和光电传感器共同探测空中低慢小目标的图像处理方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008216126A (ja) * 2007-03-06 2008-09-18 Konica Minolta Holdings Inc 距離画像生成装置、距離画像生成方法及びプログラム
JP2012008584A (ja) * 2011-08-05 2012-01-12 Toshiba Corp 映像表示装置、通知方法、及び位置判定装置
JP2012212428A (ja) * 2011-03-23 2012-11-01 Sharp Corp 視差算出装置、距離算出装置及び視差算出方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
SG64486A1 (en) * 1997-03-27 1999-04-27 Sony Corp Method and apparatus for information processing computer readable medium and authoring system
JP4005679B2 (ja) * 1997-11-19 2007-11-07 富士重工業株式会社 自律走行車の周囲環境認識装置
JP2001346031A (ja) * 2000-06-05 2001-12-14 Fuji Photo Film Co Ltd 画像合成方法及び装置
EP1946203A2 (en) 2005-10-26 2008-07-23 Sony Computer Entertainment America, Inc. System and method for interfacing with a computer program
JP2008309637A (ja) 2007-06-14 2008-12-25 Konica Minolta Holdings Inc 障害物計測方法、障害物計測装置及び障害物計測システム
JP2012002683A (ja) * 2010-06-17 2012-01-05 Fuji Electric Co Ltd ステレオ画像処理方法およびステレオ画像処理装置
JP6186184B2 (ja) * 2013-06-21 2017-08-23 株式会社ソニー・インタラクティブエンタテインメント 画像処理装置、画像処理システム、および画像処理方法
WO2019082797A1 (ja) * 2017-10-23 2019-05-02 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 再構成方法および再構成装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008216126A (ja) * 2007-03-06 2008-09-18 Konica Minolta Holdings Inc 距離画像生成装置、距離画像生成方法及びプログラム
JP2012212428A (ja) * 2011-03-23 2012-11-01 Sharp Corp 視差算出装置、距離算出装置及び視差算出方法
JP2012008584A (ja) * 2011-08-05 2012-01-12 Toshiba Corp 映像表示装置、通知方法、及び位置判定装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3343500A4 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019082797A1 (ja) * 2017-10-23 2019-05-02 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 再構成方法および再構成装置
JPWO2019082797A1 (ja) * 2017-10-23 2020-11-12 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 再構成方法および再構成装置
JP7105246B2 (ja) 2017-10-23 2022-07-22 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 再構成方法および再構成装置
JP2024020074A (ja) * 2022-08-01 2024-02-14 トヨタ自動車株式会社 情報処理方法及び情報処理装置
JP7715095B2 (ja) 2022-08-01 2025-07-30 トヨタ自動車株式会社 情報処理方法及び情報処理装置
US12462344B2 (en) 2022-08-01 2025-11-04 Toyota Jidosha Kabushiki Kaisha Information processing method and information processing apparatus

Also Published As

Publication number Publication date
US10559089B2 (en) 2020-02-11
JP6548518B2 (ja) 2019-07-24
JP2017045283A (ja) 2017-03-02
EP3343500A4 (en) 2019-12-04
EP3343500B1 (en) 2021-06-30
US20180189975A1 (en) 2018-07-05
EP3343500A1 (en) 2018-07-04

Similar Documents

Publication Publication Date Title
JP6548518B2 (ja) 情報処理装置および情報処理方法
US9542755B2 (en) Image processor and image processing method
US10293252B2 (en) Image processing device, system and method based on position detection
EP2590396B1 (en) Information processing system and information processing method
JP2014106732A (ja) 情報処理装置および情報処理方法
US9148637B2 (en) Face detection and tracking
US9767611B2 (en) Information processing apparatus and method for estimating depth values using an approximate plane
US20130335535A1 (en) Digital 3d camera using periodic illumination
CN104380338A (zh) 信息处理器以及信息处理方法
WO2015122079A1 (ja) 情報処理装置および情報処理方法
US10063838B2 (en) Information processor, information processing method, and computer program
US9785854B2 (en) Information processor, information processing method, and computer program
CN110998659A (zh) 图像处理系统、图像处理方法、及程序
US11138743B2 (en) Method and apparatus for a synchronous motion of a human body model
CN110678905B (zh) 用于处理深度图的装置及方法
JP6300560B2 (ja) 情報処理装置および情報処理方法
KR20120092960A (ko) 가상 캐릭터 제어 시스템 및 방법
JP6103942B2 (ja) 画像データ処理装置及び画像データ処理プログラム
JP7326965B2 (ja) 画像処理装置、画像処理プログラム、及び画像処理方法
JP6165650B2 (ja) 情報処理装置および情報処理方法
WO2016181672A1 (ja) 画像解析装置、画像解析方法、及び、画像解析プログラム
JP2008090608A (ja) 画像処理装置、画像処理装置の制御方法、および画像処理装置の制御プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16839200

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE