US20130163854A1 - Image processing method and associated apparatus - Google Patents

Image processing method and associated apparatus Download PDF

Info

Publication number
US20130163854A1
US20130163854A1 US13/335,952 US201113335952A US2013163854A1 US 20130163854 A1 US20130163854 A1 US 20130163854A1 US 201113335952 A US201113335952 A US 201113335952A US 2013163854 A1 US2013163854 A1 US 2013163854A1
Authority
US
United States
Prior art keywords
images
image
visual effect
alignment
processing method
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/335,952
Other languages
English (en)
Inventor
Chia-Ming Cheng
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
MediaTek Inc
Original Assignee
MediaTek Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by MediaTek Inc filed Critical MediaTek Inc
Priority to US13/335,952 priority Critical patent/US20130163854A1/en
Assigned to MEDIATEK INC. reassignment MEDIATEK INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHENG, CHIA-MING
Priority to TW101131532A priority patent/TWI544448B/zh
Priority to CN201510584676.4A priority patent/CN105187814B/zh
Priority to CN201210326621.XA priority patent/CN103179413B/zh
Publication of US20130163854A1 publication Critical patent/US20130163854A1/en
Priority to US14/300,243 priority patent/US9123125B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/18—Image warping, e.g. rearranging pixels individually
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/221—Image signal generators using stereoscopic image cameras using a single 2D image sensor using the relative movement between cameras and objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/296—Synchronisation thereof; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2625—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect
    • H04N5/2627—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect for providing spin image effect, 3D stop motion effect or temporal freeze effect

Definitions

  • the present invention relates to three-dimensional (3D) visual effect reproduction, and more particularly, to an image processing method, and to an associated apparatus.
  • 3D visual effect reproduction typically requires preparation of source images and complicated calculations.
  • the source images should be captured from a plurality of pre-calibrated cameras, where the pre-calibrated cameras should have been calibrated with respect to predetermined view points or predetermined lines of views, which causes difficulty of the preparation of the source images.
  • it is required to prepare a high end computer having high calculation power, where the high end computer would never be replaced by a conventional multifunctional mobile phone since it seems unlikely that the conventional multifunctional mobile phone can work well under the heavy calculation load of the complicated calculations.
  • the conventional multifunctional mobile phone can never be a total solution to 3D production/reproduction.
  • the related art does not serve the end user well.
  • a novel method is required for performing image processing regarding 3D visual effect reproduction in a smart and robust manner, in order to implement the preparation of the source images mentioned above and associated calculations within a portable electronic device such as a multifunctional mobile phone.
  • a portable electronic device e.g. a mobile phone, a laptop computer, or a tablet.
  • An exemplary embodiment of an image processing method comprises: receiving image data of a plurality of images, the images being captured under different view points (or along different lines of views); and performing image alignment for the plurality of images by warping the plurality of images according to the image data, wherein the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters.
  • An exemplary embodiment of an apparatus for performing image processing comprises at least one portion of an electronic device.
  • the apparatus comprises: a storage and a processing circuit.
  • the storage is arranged to temporarily store information.
  • the processing circuit is arranged to control operations of the electronic device, to receive image data of a plurality of images, the images being captured under different view points (or along different lines of views), to temporarily store the image data into the storage, and to perform image alignment for the plurality of images by warping the plurality of images according to the image data, wherein the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters.
  • FIG. 1 is a diagram of an apparatus for performing image processing according to a first embodiment of the present invention.
  • FIG. 2 illustrates the apparatus shown in FIG. 1 according to an embodiment of the present invention, where the apparatus of this embodiment is a mobile phone.
  • FIG. 3 illustrates the apparatus shown in FIG. 1 according to another embodiment of the present invention, where the apparatus of this embodiment is a personal computer such as a laptop computer.
  • FIG. 4 illustrates a flowchart of an image processing method according to an embodiment of the present invention.
  • FIG. 5 illustrates an input image and some transformation images generated during a learning/training procedure involved with the image processing method shown in FIG. 4 according to an embodiment of the present invention, where the learning/training procedure is utilized for determining a predefined solution space.
  • FIG. 6 illustrates some images obtained from multi-view vertical alignment involved with the image processing method shown in FIG. 4 according to an embodiment of the present invention.
  • FIG. 7 illustrates one of the images obtained from the multi-view vertical alignment and an associated image obtained from horizontal alignment involved with the image processing method shown in FIG. 4 according to an embodiment of the present invention.
  • FIG. 8 illustrates another of the images obtained from the multi-view vertical alignment and an associated image obtained from the horizontal alignment according to the embodiment shown in FIG. 7 .
  • FIG. 9 illustrates another of the images obtained from the multi-view vertical alignment and an associated image obtained from the horizontal alignment according to the embodiment shown in FIG. 7 .
  • FIG. 10 illustrates a disparity histogram involved with the image processing method shown in FIG. 4 according to an embodiment of the present invention.
  • FIG. 11 illustrates the apparatus shown in FIG. 1 according to an embodiment of the present invention, where the processing circuit thereof comprises some processing modules involved with the image processing method shown in FIG. 4 , and can selectively operate with aid of motion information generated by some motion sensors when needed.
  • FIG. 12 illustrates two images under processing of global/local coordinate transformation involved with the image processing method shown in FIG. 4 according to an embodiment of the present invention.
  • FIG. 13 illustrates the global/local coordinate transformation performed on the two images shown in FIG. 12 .
  • FIG. 14 illustrates two images aligned to background motion(s) for three-dimensional (3D) display involved with the image processing method shown in FIG. 4 according to an embodiment of the present invention.
  • FIG. 15 illustrates two images aligned to foreground motion(s) for a multi-angle view (MAV) visual effect involved with the image processing method shown in FIG. 4 according to another embodiment of the present invention.
  • MAV multi-angle view
  • FIG. 16 illustrates a portrait mode of a plurality of display modes involved with the image processing method shown in FIG. 4 according to an embodiment of the present invention.
  • FIG. 17 illustrates a panorama mode of the plurality of display modes according to the embodiment shown in FIG. 16 .
  • the apparatus 100 may comprise at least one portion (e.g. a portion or all) of an electronic device.
  • the apparatus 100 may comprise a portion of the electronic device mentioned above, and more particularly, can be a control circuit such as an integrated circuit (IC) within the electronic device.
  • the apparatus 100 can be the whole of the electronic device mentioned above.
  • the apparatus 100 can be an audio/video system comprising the electronic device mentioned above.
  • the electronic device may include, but not limited to, a mobile phone (e.g.
  • a multifunctional mobile phone a personal digital assistant (PDA), a portable electronic device such as the so-called tablet (based on a generalized definition), and a personal computer such as a tablet personal computer (which can also be referred to as the tablet, for simplicity), a laptop computer, or desktop computer.
  • PDA personal digital assistant
  • tablet based on a generalized definition
  • personal computer such as a tablet personal computer (which can also be referred to as the tablet, for simplicity), a laptop computer, or desktop computer.
  • the apparatus 100 comprises a processing circuit 110 and a storage 120 .
  • the storage 120 is arranged to temporarily store information, such as information carried by at least one input signal 108 that is inputted into the processing circuit 110 .
  • the storage 120 can be a memory (e.g. a volatile memory such as a random access memory (RAM), or a non-volatile memory such as a Flash memory), or can be a hard disk drive (HDD).
  • RAM random access memory
  • HDD hard disk drive
  • the processing circuit 110 is arranged to control operations of the electronic device, to receive image data of a plurality of images, the images being captured under different view points (or along different lines of views), to temporarily store the image data into the storage 120 , and to perform image alignment for the plurality of images by warping the plurality of images according to the image data, where the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters.
  • the images are captured, and more particularly, are arbitrarily captured under different view points by utilizing a camera module of the electronic device mentioned above, where the aforementioned image data can be received through the input signal 108 that is input into the processing circuit 110 .
  • the images are captured, and more particularly, are arbitrarily captured under different view points by utilizing an external device such as a hand-held camera.
  • the camera module mentioned above is not calibrated with regard to the view points (or the lines of views) mentioned above.
  • the processing circuit 110 automatically performs the image alignment to reproduce a three-dimensional (3D) visual effect, and more particularly, generates 3D images to reproduce the 3D visual effect, where the 3D images may comprise emulated images that are not generated by utilizing any camera such as the camera module mentioned above.
  • the processing circuit 110 may output information of the 3D images through at least one output signal 128 that carries the information of the 3D images.
  • a screen of the electronic device can be utilized for displaying animation based upon the 3D images to reproduce the 3D visual effect.
  • the screen may provide the user with stereoscopic views based upon the 3D images to reproduce the 3D visual effect.
  • examples of the 3D visual effect may comprise (but not limited to) a multi-angle view (MAV) visual effect and a 3D panorama visual effect.
  • the apparatus 100 can output the information of the 3D images, in order to reproduce the 3D visual effect by utilizing an external display device.
  • MAV multi-angle view
  • FIG. 2 illustrates the apparatus 100 shown in FIG. 1 according to an embodiment of the present invention, where the apparatus 100 of this embodiment is a mobile phone, and therefore, is labeled “Mobile phone” in FIG. 2 .
  • a camera module 130 (labeled “Camera” in FIG. 2 , for brevity) is taken as an example of the camera module mentioned in the first embodiment, and is installed within the apparatus 100 mentioned above (i.e. the mobile phone in this embodiment), which means the apparatus 100 comprises the camera module 130 .
  • the camera module 130 is positioned around an upper side of the apparatus 100 . This is for illustrative purposes only, and is not meant to be a limitation of the present invention.
  • the camera module 130 can be positioned around another side of the apparatus 100 .
  • a touch screen 150 (labeled “Screen” in FIG. 2 , for brevity) is taken as an example of the screen mentioned in the first embodiment, and is installed within the apparatus 100 mentioned above, which means the apparatus 100 comprises the touch screen 150 .
  • the camera module 130 can be utilized for capturing the plurality of images mentioned above.
  • the processing circuit 110 can perform feature extraction and feature matching to determine/find out the aforementioned solution constrained to the predetermined ranges of physical camera parameters, such as some predetermined ranges of physical parameters of the camera module 130 (e.g. directions/angles of the lines of views of the camera module 130 ).
  • the processing circuit 110 can generate the 3D images bounded to the aforementioned solution, in order to reproduce the 3D visual effect without introducing visible artifacts.
  • FIG. 3 illustrates the apparatus 100 shown in FIG. 1 according to another embodiment of the present invention, where the apparatus 100 of this embodiment is a personal computer such as a laptop computer, and therefore, is labeled “Laptop computer” in FIG. 3 .
  • the camera module 130 (labeled “Camera” in FIG. 3 , for brevity) is taken as an example of the camera module mentioned in the first embodiment, and is installed within the apparatus 100 mentioned above (i.e. laptop computer in this embodiment), which means the apparatus 100 comprises the camera module 130 .
  • the camera module 130 is positioned around an upper side of the apparatus 100 . This is for illustrative purposes only, and is not meant to be a limitation of the present invention.
  • the camera module 130 can be positioned around another side of the apparatus 100 .
  • a screen 50 e.g. a liquid crystal display (LCD) panel
  • LCD liquid crystal display
  • FIG. 4 illustrates a flowchart of an image processing method 200 according to an embodiment of the present invention.
  • the image processing method 200 shown in FIG. 4 can be applied to the apparatus 100 shown in FIG. 1 , and more particualrly, the apparatus 100 of any of the embodiments respectively shown in FIG. 3 and FIG. 4 .
  • the image processing method 200 is described as follows.
  • the processing circuit 110 receives image data of a plurality of images, the images being captured under different view points (e.g., the plurality of images disclosed in the first embodiment).
  • the aforementioned image data can be received through the input signal 108 that is input into the processing circuit 110 .
  • the images are captured under these different view points (or along different lines of views), and more particularly, are arbitrarily captured by utilizing a camera module such as the camera module 130 disclosed above. Please note that it is unnecessary for the camera module mentioned above to be calibrated. More particularly, the camera module of this embodiment is not calibrated with regard to the view points.
  • the processing circuit 110 performs image alignment for the plurality of images by warping the plurality of images according to the image data, where the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters.
  • the image alignment may include vertical alignment and horizontal alignment, where the horizontal alignment is typically performed after the vertical alignment is performed.
  • the horizontal alignment can be performed under disparity analysis, where the disparity analysis is utilized for analyzing warped images of the vertical alignment. This is for illustrative purposes only, and is not meant to be a limitation of the present invention.
  • the preparation/beginning of the horizontal alignment can be performed after the preparation/beginning of the vertical alignment is performed, and the horizontal alignment and the vertical alignment can be completed at the same time when some warping operations are completed.
  • the processing circuit 110 is preferably arranged to determine the aforementioned predetermined ranges of physical camera parameters in advance by performing operations of sub-steps (1), (2), (3), (4), and (5) as follows:
  • the processing circuit 110 controls the camera module 130 to capture a base image, such as one of the plurality of images mentioned in Step 210 ; (2) the processing circuit 110 controls the camera module 130 to capture multiple reference images, such as others within the plurality of images mentioned in Step 210 ; (3) the processing circuit 110 records one set of physical camera parameters corresponding to each reference image, where the aforementioned one set of physical camera parameters can be some location/coordinate-related physical parameters of the camera module 130 disclosed above (e.g. directions/angles of the lines of views of the camera module 130 ), and may comprise some physical parameters that are not location/coordinate-related (e.g.
  • the processing circuit 110 records warps the base image to match each reference image according to the recorded set of physical camera parameters, and therefore, generates a series of warped base images corresponding to each reference image; and (5) the processing circuit 110 determines the aforementioned predetermined ranges of physical camera parameters by finding whether difference(s) between warped base images and the reference images is distinguishable under human vision, where the criterion (or criteria) for determining whether the difference(s) is distinguishable under human vision or not can be predefined based upon some predefined rules.
  • the processing circuit 110 eventually determines the aforementioned predetermined ranges of physical camera parameters, in order to achieve better performance during the operations disclosed in FIG. 4 .
  • the solution constrained to the aforementioned predetermined ranges of physical camera parameters i.e. the solution mentioned in the descriptions for Step 220
  • the solution allows the base image to be arbitrarily warped while the associated physical camera parameters of this arbitrarily warping operation keep falling within the aforementioned predetermined ranges of physical camera parameters.
  • the aforementioned predetermined ranges of physical camera parameters is preferably determined by finding whether any difference between warped base images and the reference images is distinguishable under human vision, the solution guarantees that this arbitrarily warping operation will not cause any artifact that is distinguishable under human vision. Therefore, no artifact will be found.
  • the sub-steps (1), (2), (3), (4), and (5) are taken as examples of the operations of determining the aforementioned predetermined ranges of physical camera parameters. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, it is unnecessary to perform all of the sub-steps (1), (2), (3), (4), and (5). According to some variations of this embodiment, other sub-step(s) may be included.
  • FIG. 5 illustrates an input image 500 and some transformation images 512 , 514 , 522 , and 532 generated during a learning/training procedure involved with the image processing method 200 shown in FIG. 4 according to an embodiment of the present invention, where the learning/training procedure is utilized for determining a predefined solution space (e.g. a pre-trained solution space).
  • a predefined solution space e.g. a pre-trained solution space
  • the processing circuit 110 performs similarity transformation on the input image 500 by performing a plurality of warping operations to generate the transformation images 512 , 514 , 522 , and 532 , in order to find out the solution mentioned in the descriptions for Step 220 .
  • some of these warping operations performed during the learning/training procedure may cause visible artifacts, which are allowed during the learning/training procedure.
  • the criterion (or criteria) for determining whether the difference mentioned in the sub-step (5) of the embodiment shown in FIG. 4 is distinguishable under human vision or not is predefined based upon some predefined rules, and therefore, the solution mentioned in the descriptions for Step 220 can be referred to as the predefined solution space.
  • the processing circuit 110 provides the user with an interface, allowing the user to determine whether a transformation image under consideration (e.g. one of the transformation images 512 , 514 , 522 , and 532 ) has any artifact that is distinguishable under human vision.
  • a transformation image under consideration e.g. one of the transformation images 512 , 514 , 522 , and 532
  • the processing circuit 110 expands the predefined solution space (e.g., the predefined solution space is expanded to include the ranges of physical camera parameters corresponding to the transformation image under consideration); otherwise, the processing circuit 110 shrinks the predefined solution space (e.g., the predefined solution space is shrunk to exclude the ranges of physical camera parameters corresponding to the transformation image under consideration).
  • the input image 500 can be the base image mentioned in the sub-step (1) of the embodiment shown in FIG. 4 . This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, the input image 500 can be one of the reference images mentioned in the sub-step (2) of the embodiment shown in FIG. 4 .
  • the processing circuit 110 eventually determines the aforementioned predetermined ranges of physical camera parameters, where any warped image that is bounded within the predefined solution space will not have any artifact that is distinguishable under human vision.
  • the similarity transformation corresponding to the transformation image 522 can be considered to be visually insensible 3D similarity transformation with regard to physical camera parameters, where “visually insensible” typically represents “deformation of warped image is hard to be distinguished by human vision”. Similar descriptions are not repeated in detail for this embodiment.
  • the processing circuit 110 is capable of performing visually insensible image warping.
  • the image alignment mentioned in the descriptions for Step 220 e.g. the vertical alignment and the horizontal alignment
  • the associated image warping if any, for reproducing the 3D visual effect mentioned above
  • FIG. 6 illustrates some images 612 , 614 , and 616 obtained from multi-view vertical alignment involved with the image processing method 200 shown in FIG. 4 according to an embodiment of the present invention.
  • the multi-view vertical alignment disclosed in FIG. 6 is taken as an example of the vertical alignment mentioned in the descriptions for Step 220 .
  • some video objects respectively shown in the images 612 , 614 , and 616 are the same object in the real world.
  • each of the images 612 , 614 , and 616 comprises a partial image of the same person and further comprises a partial image of the same logo (which is illustrated with a warped shape of “LOGO”).
  • the processing circuit 110 performs feature extraction on each of the images 612 , 614 , and 616 and performs feature matching for the images 612 , 614 , and 616 to find out some common feature points in each of the images 612 , 614 , and 616 , such as the feature points illustrated with small circles on the three dashed lines crossing the images 612 , 614 , and 616 within FIG.
  • one of the common feature points can be located at the upper right corner of the logo, another of the common feature points can be located at the lower left corner of the logo, and another of the common feature points can be located at a junction of something worn by the person.
  • the processing circuit 110 aligns the images 612 , 614 , and 616 by performing rotating and/or shifting operations of their original images, which are multi-view images respectively corresponding to three view points (or three lines of views) and are a portion of the plurality of images mentioned in Step 210 in this embodiment.
  • the common feature points in the images 612 , 614 , and 616 are aligned to the same vertical locations (or the same horizontal lines such as the three dashed lines shown in FIG. 6 ), respectively, where the dashed lines crossing the three images 612 , 614 , and 616 within FIG. 6 indicates the alignment results of the multi-view vertical alignment.
  • the processing circuit 110 performs optimization over geometry constraint to solve the optimal camera parameters within a predefined solution space such as that mentioned above, and more particularly, a predefined visually insensible solution space. Similar descriptions are not repeated in detail for this embodiment.
  • FIG. 7 illustrates one of the images 612 , 614 , and 616 obtained from the multi-view vertical alignment, such as the image 612 , and an associated image 622 obtained from horizontal alignment involved with the image processing method 200 shown in FIG. 4 according to an embodiment of the present invention.
  • FIG. 8 illustrates another of the images 612 , 614 , and 616 obtained from the multi-view vertical alignment, such as the image 614 , and an associated image 624 obtained from the horizontal alignment according to this embodiment.
  • FIG. 9 illustrates another of the images 612 , 614 , and 616 obtained from the multi-view vertical alignment, such as the image 616 , and an associated image 626 obtained from the horizontal alignment according to this embodiment.
  • the processing circuit 110 can perform disparity histogram analysis for two-dimensional (2D) translation of warped images. For example, the processing circuit 110 calculates the number of pixels with regard to displacement (more particularly, horizontal displacement) for each of the images 612 , 614 , and 616 , in order to generate a disparity histogram for each of the images 612 , 614 , and 616 , such as that shown in FIG. 10 . As shown in FIG.
  • the horizontal axis represents the displacement (more particularly, the horizontal displacement) of at least one pixel (e.g., a single pixel or a group of pixels) within the image under consideration in comparison with a certain image
  • the vertical axis represents the number of pixels
  • the image under consideration can be any of the images 612 , 614 , and 616
  • the aforementioned certain image can be the base image or a specific image selected from the images 612 , 614 , and 616 .
  • the processing circuit 110 performs the horizontal alignment on the image 612 by cropping a portion of the image 612 to obtain the image 622 .
  • the processing circuit 110 performs the horizontal alignment on the image 614 by cropping a portion of the image 614 to obtain the image 624 .
  • the processing circuit 110 performs the horizontal alignment on the image 616 by cropping a portion of the image 616 to obtain the image 626 . Similar descriptions are not repeated in detail for this embodiment.
  • the processing circuit 110 performs the learning/training procedure, the multi-view vertical alignment, and the horizontal alignment as disclosed above, and further performs sequence reproduction, for reproducing the 3D visual effect mentioned above.
  • the processing circuit 110 performs the sequence reproduction by generating a series of warped images ⁇ 613 - 1 , 613 - 2 , . . . ⁇ that vary from the image 612 to the image 614 and by generating a series of warped images ⁇ 615 - 1 , 615 - 2 , . . .
  • FIG. 11 illustrates the apparatus 100 shown in FIG. 1 according to an embodiment of the present invention, where the processing circuit 110 thereof comprises some processing modules involved with the image processing method 200 shown in FIG. 4 , and can selectively operate with aid of motion information generated by some motion sensors 130 when needed.
  • Examples of the processing modules mentioned above may comprise a feature extraction module 1102 (labeled “Feature extraction”), a feature matching module 1104 (labeled “Feature matching”), a trained camera parameter prior module 1112 (labeled “Trained camera parameter prior”), a multi-view vertical alignment module 1114 (labeled “Multi-view vertical alignment”), a horizontal alignment module 1116 (labeled “Horizontal alignment”), a various 3D visual effect user interface (UI) module 1118 (labeled “UI for various 3D visual effect”), and an image warping module 1122 (labeled “Image warping”).
  • a feature extraction module 1102 labeleled “Feature extraction”
  • Feature matching label matching
  • a trained camera parameter prior module 1112 labeleled “Trained camera parameter prior”
  • a multi-view vertical alignment module 1114 labeleled “Multi-view vertical alignment”
  • a horizontal alignment module 1116 labeleled “Horizontal alignment”
  • UI 3D visual effect user interface
  • UI 3D visual effect user interface
  • the feature extraction module 1102 and the feature matching module 1104 are arranged to perform the feature extraction and the feature matching disclosed above, respectively, while the trained camera parameter prior module 1112 is arranged to store results of the learning/training results regarding the aforementioned solution such as the predefined solution space, and more particularly, some trained camera parameters that are obtained during the learning/training procedure.
  • the multi-view vertical alignment module 1114 and the horizontal alignment module 1116 are arranged to perform at least one portion (e.g. a portion or all) of the multi-view vertical alignment disclosed above and at least one portion (e.g.
  • the image warping module 1122 is arranged to perform image warping (more particularly, the aforementioned visually insensible image warping) when needed.
  • the various 3D visual effect UI module 1118 is arranged to reproduce the 3D visual effect mentioned above, and more particularly, to perform various kinds of 3D visual effects when needed.
  • the motion sensors 130 can be optional since the processing circuit 110 can operate properly and correctly without the aid of the aforementioned motion information generated by the motion sensors 130 , and therefore, the information paths from the motion sensors 130 to the processing circuit 110 are illustrated with dashed lines to indicate the fact that the motion sensors 130 can be optional.
  • the calculation load of the processing circuit 110 can be decreased since the aforementioned motion information may be helpful. Similar descriptions are not repeated in detail for this embodiment.
  • FIG. 12 illustrates two images IMG 1 and IMG 2 under processing of global/local coordinate transformation involved with the image processing method 200 shown in FIG. 4 according to an embodiment of the present invention
  • FIG. 13 illustrates the global/local coordinate transformation performed on the two images IMG 1 and IMG 2 shown in FIG. 12 .
  • the processing circuit 110 can perform image processing on the images IMG 1 and IMG 2 by performing rotating, shifting, cropping, and/or warping operations on the images IMG 1 and IMG 2 , respectively, where the warped rectangles respectively illustrated in the images IMG 1 and IMG 2 shown in FIG. 12 (i.e. those depicted with non-dashed lines) may represent the processed results of the images IMG 1 and IMG 2 , respectively.
  • the warped rectangles respectively illustrated in the images IMG 1 and IMG 2 shown in FIG. 12 i.e. those depicted with non-dashed lines
  • the processing circuit 110 may determine a clipping region for each of the processed results of the images IMG 1 and IMG 2 by virtually “overlapping” the images IMG 1 and IMG 2 and the processed results thereof on a set of global coordinates, which can be regarded as a common set of global coordinates for processing the images IMG 1 and IMG 2 .
  • a set of global coordinates which can be regarded as a common set of global coordinates for processing the images IMG 1 and IMG 2 .
  • the start point for the image IMG 1 can be located at the origin, and the start point for the image IMG 2 can be located at a specific point on the horizontal axis, where the clipping region can be a maximum rectangular region available for both of the processed results of the images IMG 1 and IMG 2 .
  • the start point for the image IMG 1 and the start point for the image IMG 2 can be determined based upon the proposed distance from the eyes of a viewer (e.g. the user) to the point of focus, such as the proposed distance (more particularly, the proposed horizontal distance) between the viewer and the convergence point where the sightlines of the respective eyes of the viewer converge.
  • the processing circuit 110 can align the images IMG 1 and IMG 2 to background motion(s), where the embodiment shown in FIG. 14 is typical of this situation.
  • the start point for the image IMG 1 and the start point for the image IMG 2 may be close to each other.
  • the processing circuit 110 can align the images IMG 1 and IMG 2 to foreground motion(s), where the embodiment shown in FIG. 15 is typical of this situation.
  • the start point for the image IMG 1 and the start point for the image IMG 2 may be far from each other, in comparison with the embodiment shown in FIG. 14 .
  • FIG. 16 illustrates a portrait mode of a plurality of display modes involved with the image processing method 200 shown in FIG. 4 according to an embodiment of the present invention
  • FIG. 17 illustrates a panorama mode of the plurality of display modes according to this embodiment, where the processing circuit 110 is capable of switching between different display modes within the plurality of display modes, and more particularly, is capable of switching between the portrait mode and the panorama mode.
  • some warped images bounded to the aforementioned predefined solution space are aligned to foreground, such as the location where an actor/actress is supposed to be in front of the viewer.
  • the processing circuit 110 rearrange these warped images in a reversed order and utilizes the rearranged warped images as the warped images for the panorama mode, where the leftmost warped image shown in FIG. 16 (i.e. the warped image IMG 11 thereof) is arranged to be the rightmost warped image shown in FIG.
  • the processing circuit 110 rearrange the warped images for the panorama mode in a reversed order and utilizes the rearranged warped images as the warped images for the portrait mode. Similar descriptions are not repeated in detail for this variation.
  • the image processing method 200 comprises performing automatic multiple image alignment in terms of the aforementioned predefined solution space (e.g. the pre-trained solution space) for reproducing the 3D visual effect from an image sequence, in which each image can be captured with an uncalibrated camera module (e.g. the camera module 130 ) or an uncalibrated hand-held camera. More particularly, the image processing method 200 comprises learning a model consisted of physical camera parameters, which can be utilized for performing visually insensible image warping. Examples of the parameters under consideration may comprise the extrinsic parameters for rotational variations, the intrinsic parameters for camera calibration matrix and lens distortion.
  • the image processing method 200 further comprises, from the corresponding feature points of the input sequence, performing the multi-view vertical alignment constrained by the learned camera parameters.
  • the alignment process turns out a constrained optimization problem for image warping that is visually insensible to human vision.
  • the image processing method 200 further comprises performing the horizontal alignment through the disparity analysis from the vertically-aligned matching points.
  • the image processing method 200 further comprises utilizing the UI such as a graphical UI (GUI) to reproduce the 3D visual effect by using the extracted alignment information and the warped image sequence.
  • GUI graphical UI
  • the present invention method and apparatus can generate warped images bounded to the aforementioned solution such as the predefined solution space (e.g. the pre-trained solution space) to make the associated learned geometric distortion be insensible by human vision, so that there is no artifact in respective reproduced image.
  • the optimization over the solution space according to the learned geometry constraint can always generate rational results.
  • the working flow of the associated calculations can be highly paralleled, and the associated computational complexity is low and the required memory resource is economy.
  • the present invention method and apparatus are robust to the image noises and outliers. Additionally, the present invention method and apparatus preserve the relative disparity/depth information in the warped image sequence, which is very important to image-based 3D applications.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Image Processing (AREA)
US13/335,952 2011-12-23 2011-12-23 Image processing method and associated apparatus Abandoned US20130163854A1 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US13/335,952 US20130163854A1 (en) 2011-12-23 2011-12-23 Image processing method and associated apparatus
TW101131532A TWI544448B (zh) 2011-12-23 2012-08-30 影像處理方法以及影像處理裝置
CN201510584676.4A CN105187814B (zh) 2011-12-23 2012-09-05 影像处理方法以及影像处理装置
CN201210326621.XA CN103179413B (zh) 2011-12-23 2012-09-05 影像处理方法以及影像处理装置
US14/300,243 US9123125B2 (en) 2011-12-23 2014-06-10 Image processing method and associated apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US13/335,952 US20130163854A1 (en) 2011-12-23 2011-12-23 Image processing method and associated apparatus

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US14/300,243 Continuation US9123125B2 (en) 2011-12-23 2014-06-10 Image processing method and associated apparatus

Publications (1)

Publication Number Publication Date
US20130163854A1 true US20130163854A1 (en) 2013-06-27

Family

ID=48638991

Family Applications (2)

Application Number Title Priority Date Filing Date
US13/335,952 Abandoned US20130163854A1 (en) 2011-12-23 2011-12-23 Image processing method and associated apparatus
US14/300,243 Expired - Fee Related US9123125B2 (en) 2011-12-23 2014-06-10 Image processing method and associated apparatus

Family Applications After (1)

Application Number Title Priority Date Filing Date
US14/300,243 Expired - Fee Related US9123125B2 (en) 2011-12-23 2014-06-10 Image processing method and associated apparatus

Country Status (3)

Country Link
US (2) US20130163854A1 (zh)
CN (2) CN105187814B (zh)
TW (1) TWI544448B (zh)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140002614A1 (en) * 2012-07-02 2014-01-02 Sony Pictures Technologies Inc. System and method for alignment of stereo views
US20140270693A1 (en) * 2013-03-18 2014-09-18 Nintendo Co., Ltd. Information processing device, storage medium having moving image data stored thereon, information processing system, storage medium having moving image reproduction program stored thereon, and moving image reproduction method
US20150009291A1 (en) * 2013-07-05 2015-01-08 Mediatek Inc. On-line stereo camera calibration device and method for generating stereo camera parameters
US20150326847A1 (en) * 2012-11-30 2015-11-12 Thomson Licensing Method and system for capturing a 3d image using single camera
US20160140733A1 (en) * 2014-11-13 2016-05-19 Futurewei Technologies, Inc. Method and systems for multi-view high-speed motion capture
CN111095350A (zh) * 2017-09-01 2020-05-01 三星电子株式会社 图像处理设备、用于处理图像的方法和计算机可读记录介质
CN111340015A (zh) * 2020-02-25 2020-06-26 北京百度网讯科技有限公司 定位方法和装置
US11037348B2 (en) * 2016-08-19 2021-06-15 Beijing Sensetime Technology Development Co., Ltd Method and apparatus for displaying business object in video image and electronic device
US11074752B2 (en) * 2018-02-23 2021-07-27 Sony Group Corporation Methods, devices and computer program products for gradient based depth reconstructions with robust statistics
US11120526B1 (en) * 2019-04-05 2021-09-14 Snap Inc. Deep feature generative adversarial neural networks

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ES2968868T3 (es) 2013-12-13 2024-05-14 Huawei Device Co Ltd Método y terminal para adquirir una imagen panorámica
US9883118B2 (en) 2014-03-24 2018-01-30 Htc Corporation Method of image correction and image capturing device thereof
US9436278B2 (en) 2014-04-01 2016-09-06 Moju Labs, Inc. Motion-based content navigation
CN106792223A (zh) * 2016-12-31 2017-05-31 天脉聚源（北京）科技有限公司 互动界面中多画面播放的方法和装置
CN107050859B (zh) * 2017-04-07 2020-10-27 福州智永信息科技有限公司 一种基于unity3D的拖动相机在场景中位移的方法
CN107610179B (zh) * 2017-09-04 2021-01-05 苏州佳世达电通有限公司 一种影像撷取装置的校正方法
CN109785390B (zh) * 2017-11-13 2022-04-01 虹软科技股份有限公司 一种用于图像矫正的方法和装置
CN109785225B (zh) * 2017-11-13 2023-06-16 虹软科技股份有限公司 一种用于图像矫正的方法和装置
TWI720447B (zh) * 2019-03-28 2021-03-01 財團法人工業技術研究院 影像定位方法及其系統
TWI784605B (zh) * 2021-06-29 2022-11-21 倍利科技股份有限公司 影像拼接方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6674892B1 (en) * 1999-11-01 2004-01-06 Canon Kabushiki Kaisha Correcting an epipolar axis for skew and offset
US20110141227A1 (en) * 2009-12-11 2011-06-16 Petronel Bigioi Stereoscopic (3d) panorama creation on handheld device

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5850352A (en) * 1995-03-31 1998-12-15 The Regents Of The University Of California Immersive video, including video hypermosaicing to generate from multiple video views of a scene a three-dimensional video mosaic from which diverse virtual video scene images are synthesized, including panoramic, scene interactive and stereoscopic images
GB2372659A (en) * 2001-02-23 2002-08-28 Sharp Kk A method of rectifying a stereoscopic image
US20080310760A1 (en) * 2005-11-14 2008-12-18 Koninklijke Philips Electronics, N.V. Method, a System and a Computer Program for Volumetric Registration
US20070165942A1 (en) * 2006-01-18 2007-07-19 Eastman Kodak Company Method for rectifying stereoscopic display systems
JP2009077234A (ja) * 2007-09-21 2009-04-09 Toshiba Corp 三次元画像処理装置、方法及びプログラム
JP5321011B2 (ja) * 2008-11-25 2013-10-23 ソニー株式会社 画像信号処理装置、画像信号処理方法および画像投射装置
US20110279641A1 (en) * 2010-05-14 2011-11-17 New York University Apparatus, systems, computer-accessible medium and methods for video cropping, temporally-coherent warping and retargeting
US8817080B2 (en) * 2010-12-02 2014-08-26 At&T Intellectual Property I, L.P. Location based media display
US8452081B2 (en) * 2011-01-11 2013-05-28 Eastman Kodak Company Forming 3D models using multiple images

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6674892B1 (en) * 1999-11-01 2004-01-06 Canon Kabushiki Kaisha Correcting an epipolar axis for skew and offset
US20110141227A1 (en) * 2009-12-11 2011-06-16 Petronel Bigioi Stereoscopic (3d) panorama creation on handheld device

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
Cheng et al, A novel structure-from-motion strategy for refining depth map estimation and multi-view synthesis in 3DTV, July 23, 2010, In Multimedia and Expo (ICME), 2010 IEEE International Conference on, pp. 944-949 *
Farre et al, Automatic Content Creation for Multiview Autostereoscopic Displays Using Image Domain Warping, July 15, 2011, Multimedia and Expo (ICME), 2011 IEEE International Conference on, pp. 1-6 *
Lang et al, Nonlinear Disparity Mapping for Stereoscopic 3D, 2010, ACM Transactions on Graphics, Vol. 29, No. 4, Article 75, pp. 1-10 (pages are labeled 75:1 to 75:10) *
Wopking, Viewing comfort with stereoscopic pictures: An experimental study on the subjective effects of disparity magnitude and depth of focus, 1995, Journal of the SID, 3/3, pp. 101-103 *
Zhang et al, "Recovering consistent video depth maps via bundle optimization," 2008, Computer Vision and Pattern Recognition, IEEE Conference on, pp. 1-8 *

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140002614A1 (en) * 2012-07-02 2014-01-02 Sony Pictures Technologies Inc. System and method for alignment of stereo views
US9013558B2 (en) * 2012-07-02 2015-04-21 Sony Corporation System and method for alignment of stereo views
US20150326847A1 (en) * 2012-11-30 2015-11-12 Thomson Licensing Method and system for capturing a 3d image using single camera
US9509907B2 (en) * 2013-03-18 2016-11-29 Nintendo Co., Ltd. Information processing device, storage medium having moving image data stored thereon, information processing system, storage medium having moving image reproduction program stored thereon, and moving image reproduction method
US20140270693A1 (en) * 2013-03-18 2014-09-18 Nintendo Co., Ltd. Information processing device, storage medium having moving image data stored thereon, information processing system, storage medium having moving image reproduction program stored thereon, and moving image reproduction method
US9955142B2 (en) * 2013-07-05 2018-04-24 Mediatek Inc. On-line stereo camera calibration device and method for generating stereo camera parameters
CN105556960A (zh) * 2013-07-05 2016-05-04 联发科技股份有限公司 主动式立体摄影校正装置以及产生立体摄影参数的方法
US20150009291A1 (en) * 2013-07-05 2015-01-08 Mediatek Inc. On-line stereo camera calibration device and method for generating stereo camera parameters
US10033989B2 (en) 2013-07-05 2018-07-24 Mediatek Inc. Synchronization controller for multi-sensor camera device and related synchronization method
US20160140733A1 (en) * 2014-11-13 2016-05-19 Futurewei Technologies, Inc. Method and systems for multi-view high-speed motion capture
US11037348B2 (en) * 2016-08-19 2021-06-15 Beijing Sensetime Technology Development Co., Ltd Method and apparatus for displaying business object in video image and electronic device
CN111095350A (zh) * 2017-09-01 2020-05-01 三星电子株式会社 图像处理设备、用于处理图像的方法和计算机可读记录介质
US11074752B2 (en) * 2018-02-23 2021-07-27 Sony Group Corporation Methods, devices and computer program products for gradient based depth reconstructions with robust statistics
US11120526B1 (en) * 2019-04-05 2021-09-14 Snap Inc. Deep feature generative adversarial neural networks
US11657479B2 (en) 2019-04-05 2023-05-23 Snap Inc. Deep feature generative adversarial neural networks
US11900565B2 (en) 2019-04-05 2024-02-13 Snap Inc. Deep feature generative adversarial neural networks
CN111340015A (zh) * 2020-02-25 2020-06-26 北京百度网讯科技有限公司 定位方法和装置
US20210264198A1 (en) * 2020-02-25 2021-08-26 Beijing Baidu Netcom Science And Technology Co., Ltd. Positioning method and apparatus

Also Published As

Publication number Publication date
TW201327472A (zh) 2013-07-01
US20140286567A1 (en) 2014-09-25
TWI544448B (zh) 2016-08-01
CN105187814A (zh) 2015-12-23
US9123125B2 (en) 2015-09-01
CN105187814B (zh) 2017-05-17
CN103179413A (zh) 2013-06-26
CN103179413B (zh) 2015-10-28

Similar Documents

Publication Publication Date Title
US9123125B2 (en) Image processing method and associated apparatus
US9619861B2 (en) Apparatus and method for improving quality of enlarged image
CN108604373B (zh) 用于使用多个相机实施无缝缩放功能的系统和方法
US10389948B2 (en) Depth-based zoom function using multiple cameras
US11663733B2 (en) Depth determination for images captured with a moving camera and representing moving features
US8401316B2 (en) Method and apparatus for block-based compression of light-field images
JP4938093B2 (ja) ２ｄ−ｔｏ−３ｄ変換のための２ｄ画像の領域分類のシステム及び方法
US10915998B2 (en) Image processing method and device
CN102469242B (zh) 成像设备和成像方法
WO2016074620A1 (en) Parallax tolerant video stitching with spatial-temporal localized warping and seam finding
US9373187B2 (en) Method and apparatus for producing a cinemagraph
KR102149463B1 (ko) 이미지를 처리하기 위한 전자 장치 및 방법
US20120105601A1 (en) Apparatus and method for creating three-dimensional panoramic image by using single camera
US20140168371A1 (en) Image processing apparatus and image refocusing method
JP2013038454A (ja) 画像処理装置および方法、並びにプログラム
US20140347548A1 (en) Method and system for rendering an image from a light-field camera
KR101518696B1 (ko) 증강현실 컨텐츠의 생성 및 재생 시스템과, 이를 이용한 방법
US20130342735A1 (en) Image processing method and image processing apparatus for performing defocus operation according to image alignment related information
JP2019036876A (ja) 画像読取装置、画像形成装置、画像読取方法及び画像読取プログラム
CN114494824A (zh) 用于全景图像的目标检测方法、装置、设备及存储介质
JP4848748B2 (ja) 撮像装置
US11119396B1 (en) Camera system with a plurality of image sensors
JP2005217902A (ja) 画像処理装置および画像処理方法

Legal Events

Date Code Title Description
AS Assignment

Owner name: MEDIATEK INC., TAIWAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CHENG, CHIA-MING;REEL/FRAME:027438/0696

Effective date: 20111213

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION