JP2018042237A5 - - Google Patents

Download PDF

Info

Publication number
JP2018042237A5
JP2018042237A5 JP2017156642A JP2017156642A JP2018042237A5 JP 2018042237 A5 JP2018042237 A5 JP 2018042237A5 JP 2017156642 A JP2017156642 A JP 2017156642A JP 2017156642 A JP2017156642 A JP 2017156642A JP 2018042237 A5 JP2018042237 A5 JP 2018042237A5
Authority
JP
Japan
Prior art keywords
virtual viewpoint
image
pixel
image processing
interest
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2017156642A
Other languages
Japanese (ja)
Other versions
JP6974978B2 (en
JP2018042237A (en
Filing date
Publication date
Application filed filed Critical
Priority to US15/686,491 priority Critical patent/US10559095B2/en
Publication of JP2018042237A publication Critical patent/JP2018042237A/en
Publication of JP2018042237A5 publication Critical patent/JP2018042237A5/ja
Application granted granted Critical
Publication of JP6974978B2 publication Critical patent/JP6974978B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Description

本発明の目的を達成するために、例えば、本発明の画像処理装置は以下の構成を有する。すなわち、
仮想視点の位置の情報と前記仮想視点からの視線方向の情報を含む仮想視点情報を取得する取得手段と、
前記取得手段により取得された仮想視点情報により特定される仮想視点に対応する仮想視点画像の着目画素の色情報を、複数の撮像装置で撮像されることにより取得された複数の撮像画像と、前記複数の撮像画像それぞれにおける端部から前記着目画素に対応する画素までの距離に応じた重みとに基づいて決定する決定手段と、
前記決定手段により決定された色情報に基づいて、前記仮想視点画像を生成する生成手段と、
を有することを特徴とする。 To achieve the object of the present invention, for example, the image processing apparatus of the present invention has the following configuration. That is,
An acquisition means for acquiring virtual viewpoint information including information on the position of the virtual viewpoint and information on the line-of-sight direction from the virtual viewpoint , and
A plurality of captured images acquired by capturing the color information of the pixel of interest of the virtual viewpoint image corresponding to the virtual viewpoint identified by the virtual viewpoint information acquired by the acquisition means with a plurality of imaging devices , and the preceding image . A determination means for determining based on the weight according to the distance from the end of each of the plurality of captured images to the pixel corresponding to the pixel of interest.
A generation means for generating the virtual viewpoint image based on the color information determined by the determination means, and
It is characterized by having .

Claims (14)

仮想視点の位置の情報と前記仮想視点からの視線方向の情報を含む仮想視点情報を取得する取得手段と、
前記取得手段により取得された仮想視点情報により特定される仮想視点に対応する仮想視点画像の着目画素の色情報を、複数の撮像装置で撮像されることにより取得された複数の撮像画像と、前記複数の撮像画像それぞれにおける端部から前記着目画素に対応する画素までの距離に応じた重みとに基づいて決定する決定手段と、
前記決定手段により決定された色情報に基づいて、前記仮想視点画像を生成する生成手段と、
を有することを特徴とする画像処理装置。 An acquisition means for acquiring virtual viewpoint information including information on the position of the virtual viewpoint and information on the line-of-sight direction from the virtual viewpoint , and
A plurality of captured images acquired by capturing the color information of the pixel of interest of the virtual viewpoint image corresponding to the virtual viewpoint identified by the virtual viewpoint information acquired by the acquisition means with a plurality of imaging devices , and the preceding image . A determination means for determining based on the weight according to the distance from the end of each of the plurality of captured images to the pixel corresponding to the pixel of interest.
A generation means for generating the virtual viewpoint image based on the color information determined by the determination means, and
An image processing device characterized by having . 前記決定手段は、前記仮想視点画像の前記着目画素の色情報を、前記複数の撮像画像それぞれにおける前記着目画素に対応する画素の色情報と、前記着目画素に対応する画素に対して前記距離に応じて設定された重みと、に基づいて決定することを特徴とする請求項１に記載の画像処理装置。The determining means converts the color information of the pixel of interest of the virtual viewpoint image into the color information of the pixel corresponding to the pixel of interest in each of the plurality of captured images and the distance to the pixel corresponding to the pixel of interest. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined based on the weights set accordingly. 前記重みは、前記撮像画像における端部から前記着目画素に対応する画素までの距離が第１の距離である場合の方が、前記撮像画像における端部から前記着目画素に対応する画素までの距離が前記第１の距離よりも大きい第２の距離である場合よりも小さいことを特徴とする、請求項１又は２に記載の画像処理装置。 The weights, who when the distance from the end portion in the captured image to the pixel corresponding to the target pixel is a first distance, the distance from the end portion in the captured image to the pixel corresponding to the pixel of interest The image processing apparatus according to claim 1 or 2 , wherein is smaller than the case where is a second distance larger than the first distance . 前記重みは、前記撮像画像における端部から前記着目画素に対応する画素までの距離が所定の閾値を超えている場合は、前記撮像画像における端部から前記着目画素に対応する画素までの距離が所定の閾値以下である場合よりも大きいことを特徴とする、請求項１乃至３の何れか１項に記載の画像処理装置。 When the distance from the end of the captured image to the pixel corresponding to the pixel of interest exceeds a predetermined threshold value, the weight is the distance from the end of the captured image to the pixel corresponding to the pixel of interest. The image processing apparatus according to any one of claims 1 to 3, wherein the image processing apparatus is larger than the case where the threshold value is equal to or less than a predetermined threshold value . 前記重みは、
前記撮像画像における端部から前記着目画素に対応する画素までの距離が所定の閾値を超えている場合、一定の重みであり、
前記撮像画像における端部から前記着目画素に対応する画素までの距離が所定の閾値以下である場合、前記一定の重みより小さい重みである
ことを特徴とする、請求項１乃至４の何れか１項に記載の画像処理装置。 The weight is
When the distance from the end portion of the captured image to the pixel corresponding to the pixel of interest exceeds a predetermined threshold value, the weight is constant.
Any one of claims 1 to 4 , wherein when the distance from the end portion of the captured image to the pixel corresponding to the pixel of interest is equal to or less than a predetermined threshold value, the weight is smaller than the constant weight. The image processing apparatus according to the section. 前記重みは、さらに、前記仮想視点からの視線方向と、前記複数の撮像装置の視線方向とに基づくことを特徴とする請求項１乃至５の何れか１項に記載の画像処理装置。The image processing apparatus according to any one of claims 1 to 5, wherein the weight is further based on a line-of-sight direction from the virtual viewpoint and a line-of-sight direction of the plurality of image pickup devices. 前記重みは、さらに、前記仮想視点の位置と、前記複数の撮像装置の位置とに基づくことを特徴とする請求項１乃至６の何れか１項に記載の画像処理装置。The image processing apparatus according to any one of claims 1 to 6, wherein the weight is further based on the position of the virtual viewpoint and the position of the plurality of imaging devices. 前記取得手段は、前記複数の撮像装置の位置の情報と、前記複数の撮像装置の視線方向の情報とを含む撮像装置情報をさらに取得し、The acquisition means further acquires image pickup device information including position information of the plurality of image pickup devices and information in the line-of-sight direction of the plurality of image pickup devices.
前記決定手段は、前記取得手段により取得された前記仮想視点情報と前記撮像装置情報とに基づいて、仮想視点画像の色情報を決定するために用いる２以上の撮像装置を特定し、特定された前記２以上の撮像装置で撮像されることにより取得された２以上の撮像画像と、前記２以上の撮像画像それぞれにおける端部から前記着目画素に対応する画素までの距離に応じた重みとに基づいて、前記仮想視点画像の色情報を決定することを特徴とする請求項１乃至７の何れか１項に記載の画像処理装置。The determination means identifies and identifies two or more image pickup devices used for determining the color information of the virtual viewpoint image based on the virtual viewpoint information and the image pickup device information acquired by the acquisition means. Based on two or more captured images acquired by being imaged by the two or more imaging devices, and weights according to the distance from the end of each of the two or more captured images to the pixel corresponding to the pixel of interest. The image processing apparatus according to any one of claims 1 to 7, wherein the color information of the virtual viewpoint image is determined. 前記生成手段は、さらにオブジェクトの３次元モデルに基づいて、前記仮想視点画像を生成することを特徴とする請求項１乃至８の何れか１項に記載の画像処理装置。The image processing apparatus according to any one of claims 1 to 8, wherein the generation means further generates the virtual viewpoint image based on a three-dimensional model of an object. 前記決定手段は、前記仮想視点画像におけるオブジェクトの色情報を決定することを特徴とする請求項１乃至９の何れか１項に記載の画像処理装置。The image processing apparatus according to any one of claims 1 to 9, wherein the determination means determines color information of an object in the virtual viewpoint image. 前記オブジェクトは、動体であることを特徴とする請求項９又は１０に記載の画像処理装置。The image processing apparatus according to claim 9 or 10, wherein the object is a moving object. 前記オブジェクトは、時間とともに位置及び形状が変化しない物体であることを特徴とする請求項９又は１０に記載の画像処理装置。The image processing apparatus according to claim 9 or 10, wherein the object is an object whose position and shape do not change with time. 画像処理装置が行う画像処理方法であって、
仮想視点の位置の情報と前記仮想視点からの視線方向の情報を含む仮想視点情報を取得する工程と、
取得された前記仮想視点情報により特定される仮想視点に対応する仮想視点画像の着目画素の色情報を、複数の撮像装置で撮像されることにより取得された複数の撮像画像と、前記複数の撮像画像それぞれにおける端部から前記着目画素に対応する画素までの距離に応じた重みとに基づいて決定する工程と、
決定された前記色情報に基づいて、前記仮想視点画像を生成する工程と、
を有することを特徴とする画像処理方法。 This is an image processing method performed by an image processing device.
A process of acquiring virtual viewpoint information including information on the position of the virtual viewpoint and information on the line-of-sight direction from the virtual viewpoint , and
A plurality of captured images acquired by capturing the color information of the pixel of interest of the virtual viewpoint image corresponding to the virtual viewpoint identified by the acquired virtual viewpoint information by a plurality of imaging devices , and the plurality of imaging A step of determining based on the weight according to the distance from the end of each image to the pixel corresponding to the pixel of interest, and
A process of generating the virtual viewpoint image based on the determined color information, and
An image processing method characterized by having . コンピュータを、請求項１乃至１２の何れか１項に記載の画像処理装置として機能させるためのプログラム。 A program for a computer to function as the image processing equipment according to any one of claims 1 to 1 2.
JP2017156642A 2016-08-31 2017-08-14 Image processing equipment, image processing methods, and programs Active JP6974978B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/686,491 US10559095B2 (en) 2016-08-31 2017-08-25 Image processing apparatus, image processing method, and medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2016170069 2016-08-31
JP2016170069 2016-08-31

Publications (3)

Publication Number Publication Date
JP2018042237A JP2018042237A (en) 2018-03-15
JP2018042237A5 true JP2018042237A5 (en) 2020-09-10
JP6974978B2 JP6974978B2 (en) 2021-12-01

Family

ID=61626620

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2017156642A Active JP6974978B2 (en) 2016-08-31 2017-08-14 Image processing equipment, image processing methods, and programs

Country Status (1)

Country Link
JP (1) JP6974978B2 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7123736B2 (en) * 2018-10-23 2022-08-23 キヤノン株式会社 Image processing device, image processing method, and program
US11704839B2 (en) * 2019-12-17 2023-07-18 Electronics And Telecommunications Research Institute Multiview video encoding and decoding method
JP2022030985A (en) * 2020-08-07 2022-02-18 ソニーグループ株式会社 Information processing device, information processing method and information processing system
CN115512038B (en) * 2022-07-22 2023-07-18 北京微视威信息科技有限公司 Real-time drawing method for free viewpoint synthesis, electronic device and readable storage medium

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001320616A (en) * 2000-02-29 2001-11-16 Matsushita Electric Ind Co Ltd Image pickup system
JP2004537082A (en) * 2001-01-26 2004-12-09 ザクセル システムズ インコーポレイテッド Real-time virtual viewpoint in virtual reality environment
JP3695377B2 (en) * 2001-10-01 2005-09-14 日産自動車株式会社 Image composition apparatus and image composition method
JP4111834B2 (en) * 2003-01-07 2008-07-02 株式会社ソニー・コンピュータエンタテインメント Image generation method and apparatus
WO2004114224A1 (en) * 2003-06-20 2004-12-29 Nippon Telegraph And Telephone Corporation Virtual visual point image generating method and 3-d image display method and device
JP4739122B2 (en) * 2006-06-05 2011-08-03 パナソニック株式会社 In-vehicle camera image composition apparatus and image composition method
JP4801549B2 (en) * 2006-09-25 2011-10-26 パナソニック株式会社 Composite video display device and integrated circuit

Similar Documents

Publication Publication Date Title
WO2016106383A3 (en) First-person camera based visual context aware system
JP2018092580A5 (en) Image processing apparatus, image processing method, and program
JP2018116692A5 (en) Information processing equipment and information processing system
JP2018056971A5 (en)
JP2017212593A5 (en) INFORMATION PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, CONTROL METHOD OF INFORMATION PROCESSING APPARATUS, GENERATION METHOD OF VIRTUAL VIEWPOINT IMAGE, AND PROGRAM
JP2017528731A5 (en)
JP2018042237A5 (en)
JP2019159739A5 (en)
JP2018004310A5 (en)
JP2019159950A5 (en)
WO2016005948A3 (en) Augmented reality system
JP2019036790A5 (en)
TWI576647B (en) Calibration guidance system and operation method of a calibration guidance system
JP2019040229A5 (en)
JP2020086983A5 (en)
EP2919159A3 (en) Image processing device and image processing method
JP2018180654A5 (en) Information processing apparatus, information processing method, and program
JP2018207252A5 (en)
JP2018026064A5 (en)
JP2015103104A5 (en)
JP2019036791A5 (en) Image processing device and its control method, program
JP2016208306A5 (en)
JP2018101942A5 (en)
JP2016220198A5 (en)
JP2015106290A5 (en)