WO2024001342A1 - 成像畸变矫正方法及装置 - Google Patents

成像畸变矫正方法及装置 Download PDF

Info

Publication number
WO2024001342A1
WO2024001342A1 PCT/CN2023/083906 CN2023083906W WO2024001342A1 WO 2024001342 A1 WO2024001342 A1 WO 2024001342A1 CN 2023083906 W CN2023083906 W CN 2023083906W WO 2024001342 A1 WO2024001342 A1 WO 2024001342A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
human body
imaging
camera
distortion
Prior art date
Application number
PCT/CN2023/083906
Other languages
English (en)
French (fr)
Inventor
徐海
范燕平
王敏波
颜国雄
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202211044984.4A external-priority patent/CN117395518A/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2024001342A1 publication Critical patent/WO2024001342A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems

Definitions

  • the present application relates to the field of image processing technology, and in particular to an imaging distortion correction (lens distortion correction, LDC) method and device.
  • imaging distortion correction lens distortion correction, LDC
  • Wide-angle cameras are increasingly used in video conferencing scenarios. Cameras with a field of view (FOV) greater than 60 degrees can usually be called wide-angle cameras.
  • FOV field of view
  • multiple human body images in the video conferencing images captured by the wide-angle camera need to be cropped out and then spliced for display.
  • the spliced images will be inconsistent, resulting in poor display effects.
  • This application provides an imaging distortion correction method and device, which can solve the current problem of poor display effects caused by the incoordination of spliced pictures based on multiple human body imaging.
  • an imaging distortion correction method is provided. This method is applied to an image processing device, which may be, for example, a conference terminal or a video server.
  • the method includes: an image processing device acquiring a first image collected by a first camera.
  • the first image includes a first human body image located in a distortion area of the first image.
  • the distortion type of human body imaging located in the distortion area of the first image includes human body tilt and/or human body deformation.
  • the image processing device performs distortion correction on the first human body image according to the first imaging position of the first human body image in the first image to obtain a second human body image.
  • the second human body imaging satisfies the following conditions: the tilt angle of the human body is less than a preset angle, and the deformation degree of the human body is less than a preset threshold.
  • the human body tilt mentioned in this application refers to the angle between the height direction of the human body (such as the line connecting the center point of the human face and the center point of the human body) and the vertical direction.
  • the tilt angle of the human body refers to the height direction of the human body and the vertical direction. Angle from straight direction.
  • the inventor found that the degree of distortion of the human body imaging in the image is affected by the imaging position. The closer the imaging position is to the edge of the image, the more serious the distortion of the human body imaging. Therefore, by correcting the human body images at different degrees of distortion of the human body images located at different positions in the image, the human body images located at different positions in the image, which have undergone distortion correction, are subsequently spliced together, so that the spliced picture can be The deformation and tilt of the human body in different human body imaging are consistent, which improves the coordination of the spliced images and thus improves the display effect.
  • the first image is collected by the first camera using linear projection, and the distortion type of human body imaging located in the distortion area of the first image includes human body deformation.
  • the image processing equipment can first perform deformation correction projection transformation on the image obtained by linear projection to improve the human body deformation of the human body image in the image.
  • the human body imaging in the image after deformation correction projection transformation will be tilted.
  • the image processing device cuts out the tilted human body image from the image that has undergone the deformation correction projection transformation, and performs rotation correction on the tilted human body image, so that the tilt distortion of the human body imaging is also improved, and finally we get Human body imaging with basically no distortion.
  • an implementation in which the image processing device determines the second imaging position of the first human body in the second image according to the first imaging position includes: the image processing device determines the second imaging position of the first human body in the second image according to the first imaging position and the field of view of the first camera. The angle and the focal length of the first camera determine the second imaging position of the first human body image in the second image.
  • the second imaging position includes the center point position of the human face and the center point position of the human body.
  • the image processing device performs rotation correction on the first human body image cropped from the second image according to the second imaging position, including: the image processing device determines the first human body according to the position of the center point of the face and the position of the center point of the human body. The tilt angle of the human body imaged in the second image.
  • the image processing device performs rotation correction on the first human body image cropped from the second image according to the tilt angle of the first human body image in the second image.
  • the deformation correction projection transformation is Panini projection transformation or spherical projection transformation.
  • the image processing device performs distortion correction on the first human body image according to the first imaging position of the first human body image in the first image, including: the image processing device based on the pre-stored imaging
  • the first corresponding relationship between the position and the deformation degree of the human body determines the human body deformation degree of the first human body imaged in the first image according to the first imaging position, wherein the first corresponding relationship matches the field of view angle of the first camera.
  • the image processing device performs deformation correction on the first human body image cropped from the first image according to the degree of human deformation of the first human body image in the first image.
  • one or more sets of correspondences between imaging positions and human body deformation degrees can be pre-stored in the image processing device, and each set of correspondences matches a camera field of view.
  • the image processing device can determine the degree of human deformation of the human body image in the image based on the corresponding relationship between the imaging position matching the camera field of view corresponding to the image and the degree of human body deformation.
  • deformation correction is performed on the cropped human body image, which can improve the human body deformation of the human body image.
  • this implementation method does not require deformation correction projection transformation, has a small amount of calculation, and has low processing performance requirements for image processing equipment.
  • the first camera is tilted relative to the horizontal plane, and the distortion type of human body imaging located in the distortion area of the first image also includes human body tilt.
  • the first imaging position includes the position of the center point of the face and the position of the center point of the human body.
  • the image processing device performs distortion correction on the first human body image based on the first imaging position of the first human body image in the first image, and also includes:
  • the image processing device determines the tilt angle of the first human body in the first image based on the position of the center point of the human face and the position of the center point of the human body.
  • the image processing device performs rotation correction on the first human body image cropped from the first image according to the tilt angle of the first human body image in the first image.
  • the first camera is arranged tilted relative to the horizontal plane, and the distortion type of human body imaging located in the distortion area of the first image also includes human body tilt.
  • the image processing device performs distortion correction on the first human body image based on the first imaging position of the first human body image in the first image.
  • the image processing device further includes: the image processing device based on the pre-stored imaging position and the second human body tilt angle.
  • the correspondence relationship determines the human body tilt angle at which the first human body is imaged in the first image, wherein the second correspondence relationship matches the field of view angle of the first camera and the pitch angle of the first camera.
  • the image processing device performs rotation correction on the first human body image cropped from the first image according to the tilt angle of the first human body image in the first image.
  • the first camera is tilted relative to the horizontal plane, and the distortion type of human body imaging located in the distortion area of the first image also includes human body tilt.
  • the image processing device After the image processing device acquires the first image collected by the first camera, the image processing device performs trapezoidal transformation on the first image according to the pitch angle of the first camera to obtain a third image.
  • the method for performing distortion correction on the first human body image according to the first imaging position of the first human body image in the first image includes: the image processing device performs distortion correction on the first human body image according to the third position in the third image. Imaging position, distortion correction is performed on the first human body image.
  • This solution when the camera is tilted relative to the horizontal plane, the image collected by the camera will undergo trapezoidal deformation.
  • This solution first performs trapezoidal transformation on the image collected by the camera to improve the trapezoidal deformation of the image, and then further transforms the trapezoidal deformation of the image. Deformation correction of the human body image in the image can improve the display effect of the final human body image.
  • the first image is collected by the first camera using Panini projection
  • the distortion types of the human body imaging located in the distortion area of the first image include human body tilt and human body deformation.
  • the image processing device performs distortion correction on the first human body image according to the first imaging position of the first human body image in the first image, including: the image processing device performs distortion correction on the first human body image according to the field of view angle of the first camera and the focal length of the first camera. , perform spherical projection transformation on the first image to obtain the fourth image.
  • the image processing device determines a fourth imaging position in the fourth image where the first human body is imaged based on the first imaging position.
  • the image processing device performs rotation correction on the first human body image cropped from the fourth image according to the fourth imaging position.
  • the first image is collected by the first camera using a spherical projection method, and the distortion type of human body imaging located in the distortion area of the first image includes human body tilt.
  • the image processing device performs distortion correction on the first human body image according to the first imaging position in the first image, including: the image processing device performs distortion correction on the first human body image according to the first imaging position.
  • the first human body imaging to perform rotation correction.
  • the image processing device can also acquire a third human body image.
  • the third human body image is a human body image cropped from the non-distorted area of the target image, or the third human body image is located in the distortion area of the target image and passes through Distortion-corrected human imaging.
  • the image processing device splices the second human body image and the third human body image to obtain a spliced image.
  • the image processing device outputs the stitched image for display on the screen.
  • the image processing device outputs the spliced image for display on the screen. It may display the spliced image on its own screen, or it may send the spliced image to other devices for display by other devices.
  • the conference terminal can splice and display the human body images of multiple participants based on the images collected by the local end. In this way, participants with scattered seats can be displayed together in the spliced screen.
  • the conference terminal can also send spliced images to other remote conference terminals for display by other conference terminals.
  • the above-mentioned target image is a first image
  • the above-mentioned target image is an image collected by a second camera
  • the second camera is different from the first camera. That is, the first human body imaging and the third human body imaging come from the same image, or the first human body imaging and the third human body imaging come from images collected by different cameras.
  • the image processing device can splice different human body images from the same image. Since the human body images used for splicing have been distorted or have no distortion in the original image, after splicing multiple human body images together, , can make the human body deformation and human body tilt degree of different human body images in the spliced picture basically consistent, and the coordination of the spliced picture is better, which can ensure the display effect.
  • the image processing device can also splice human body imaging from different images. For example, in a video conference scenario, multiple conference terminals collect video conference images at the same time, and the image processing device can stitch human body imaging from different video conference images. Splicing, by putting people in different conference rooms into the same spliced screen, can make multiple participants feel like they are working together, thus improving the multi-person collaboration atmosphere.
  • the image processing device forms an image of the first human body according to the first imaging position of the first human body in the first image.
  • An implementation method of image distortion correction includes: in response to receiving a selection instruction for a target display mode, the image processing device performs distortion correction on the first human body image according to the first imaging position.
  • the image processing device splices the second human body imaging and the third human body imaging to obtain the spliced image, including: performing a scaling operation on the second human body imaging and/or the third human body imaging according to the target display mode, and performing a zoom operation on the second human body imaging and/or the third human body imaging.
  • the second human body imaging and the third human body imaging that have been zoomed are spliced to obtain a spliced image.
  • the target display mode can be smart equalization mode or gallery mode.
  • the image processing device is a conference terminal, and the first camera is built in the image processing device, or the first camera is connected to the image processing device.
  • the first image is the image collected by the local end.
  • the field of view of the first camera and the focal length of the first camera can be sent to the image processing device by other devices.
  • the image processing device can estimate the field of view of the first camera used to capture the first image based on the first image, and then estimate the focal length of the first camera based on the estimated field of view.
  • the image processing device can use a linear regression algorithm to predict the camera field of view corresponding to the image based on AlexNet (a deep convolutional neural network).
  • AlexNet a deep convolutional neural network
  • an imaging distortion correction device in a second aspect, includes multiple functional modules, and the multiple functional modules interact to implement the method in the above-mentioned first aspect and its various implementations.
  • the multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on specific implementation.
  • an imaging distortion correction device including: a processor and a memory;
  • the memory is used to store a computer program, the computer program includes program instructions;
  • the processor is configured to call the computer program to implement the method in the above first aspect and its various implementations.
  • a computer-readable storage medium In a fourth aspect, a computer-readable storage medium is provided. Instructions are stored on the computer-readable storage medium. When the instructions are executed by a processor, the methods in the above-mentioned first aspect and its various embodiments are implemented.
  • a chip in a sixth aspect, includes programmable logic circuits and/or program instructions. When the chip is run, the method in the first aspect and its various embodiments is implemented.
  • Figure 1 is a schematic diagram of an image collected by a wide-angle camera using linear projection according to an embodiment of the present application
  • Figure 2 is a schematic diagram of another image collected by a wide-angle camera using linear projection according to an embodiment of the present application
  • Figure 3 is a schematic diagram of an image collected by a wide-angle camera using spherical projection according to an embodiment of the present application
  • Figure 4 is a schematic diagram of an image collected by a wide-angle camera using Panini projection according to an embodiment of the present application. picture;
  • Figure 5 is a schematic diagram of a spliced picture of human body imaging provided by an embodiment of the present application.
  • Figure 6 is a schematic diagram of an application scenario provided by the embodiment of the present application.
  • Figure 7 is a schematic diagram of the regional distribution of images collected by a wide-angle camera provided in an embodiment of the present application.
  • Figure 8 is a schematic flow chart of an imaging distortion correction method provided by an embodiment of the present application.
  • Figure 9 is a schematic diagram of an image provided by an embodiment of the present application.
  • Figure 10 is a schematic diagram of an image coordinate transformation provided by an embodiment of the present application.
  • Figure 11 is a schematic diagram of rotation correction for human body imaging in the second image provided by an embodiment of the present application.
  • Figure 12 is a schematic diagram of spliced images in an intelligent equalization mode provided by an embodiment of the present application.
  • Figure 13 is a schematic diagram of spliced images in gallery mode provided by an embodiment of the present application.
  • Figure 14 is a schematic diagram of spliced images in another gallery mode provided by an embodiment of the present application.
  • Figure 16 is a schematic structural diagram of an imaging distortion correction device provided by an embodiment of the present application.
  • Figure 17 is a schematic structural diagram of another imaging distortion correction device provided by an embodiment of the present application.
  • Figure 18 is a schematic diagram of the hardware structure of an imaging distortion correction device provided by an embodiment of the present application.
  • this projection method can make the image captured by the wide-angle camera
  • the horizontal lines in are straight and parallel to the ground
  • the vertical lines are straight and perpendicular to the ground.
  • this projection method will also cause the foreground image distortion closer to the edge of the image to be more serious.
  • the larger the field of view of a wide-angle camera using linear projection the more serious the foreground image distortion at the edge of the image will be.
  • FIG. 1 the larger the field of view of a wide-angle camera using linear projection, the more serious the foreground image distortion at the edge of the image will be.
  • image A includes a human body image A1 located in the edge area of the image and a human body image A2 located in the center area of the image. It is assumed that the human body posture and human body shape of the acquisition object of human body imaging A1 and the acquisition object of human body imaging A2 are different, but the standing positions are different. Referring to Figure 1, human body image A1 is obviously stretched in the horizontal direction (transverse stretching) compared to human body image A2.
  • FIG. 2 is a schematic diagram of another image collected by a wide-angle camera using linear projection according to an embodiment of the present application.
  • image B includes the human body image B1 located in the right edge area of the image and the human body image B1 located in the image Human body imaging of central area B2.
  • human body posture and human body shape of the acquisition object of human body imaging B1 and the acquisition object of human body imaging B2 are different, but the standing positions are different.
  • human body image B1 is laterally stretched compared to human body image B2, and the upper body of human body image B1 is obviously tilted to the right.
  • the tilt of the human body mentioned in the embodiments of this application means that there is a certain angle between the height direction of the human body (for example, the line connecting the center point of the human face and the center point of the human body) and the vertical direction.
  • FIG. 3 is a schematic diagram of an image collected by a wide-angle camera using spherical projection according to an embodiment of the present application.
  • image C includes a human body image C1 located in the right edge area of the image and a human body image C2 located in the center area of the image.
  • Panini projection is a user-defined projection method.
  • the image effect of Panini projection is between the image effect of linear projection and the image effect of spherical projection.
  • different degrees of distortion correction are performed according to different imaging positions in the image.
  • the main principle is to take the center point of the image as the center of the circle and the pixels on the circle with radius r, according to the actual display effect, uniformly move along the radius r and move a distance d inside or outside the circle.
  • the representation in encoding is a mapping table.
  • the mapping table is usually customized by the user according to the display effect.
  • FIG. 4 is a schematic diagram of an image collected by a wide-angle camera using Panini projection according to an embodiment of the present application.
  • image D includes a human body image D1 located in the right edge area of the image and a human body image D2 located in the center area of the image. It is assumed that the human body posture and human body shape of the acquisition object of human body imaging D1 and the acquisition object of human body imaging D2 are different, but the standing positions are different.
  • the human body image D1 is laterally stretched compared to the human body image D2, but the stretching degree is lower than the stretching degree of the human body image A1 in image A.
  • the upper body of human body image D1 is slightly tilted to the left, but the degree of tilt is lower than that of human body image C1 in image C.
  • the human body image B1 and the human body image B2 in the image B are cut and spliced together to obtain a spliced picture as shown in Figure 5, that is, the spliced picture includes image B Human body imaging B1 and human body imaging B2 in . Since the human body image B1 is laterally stretched compared to the human body image B2, and the upper body of the human body image B1 is obviously tilted to the right, the spliced picture is obviously uncoordinated and the display effect is poor.
  • embodiments of the present application provide an imaging distortion correction method, which is applied to image processing equipment.
  • the image processing device After acquiring the image collected by the camera, the image processing device performs distortion correction on the human body image according to its imaging position in the image. Since the degree of distortion of the human body imaging in the image is affected by the imaging position, the closer the imaging position is to the edge of the image, the more serious the degree of distortion of the human body imaging.
  • the human body images at different degrees of distortion of the human body images at different positions in the image the human body images at different positions in the image that have been distorted and corrected can be spliced together to achieve different human bodies in the spliced picture.
  • the imaged human body deformation and human body tilt are consistent, which improves the coordination of the spliced images and thus improves the display effect.
  • the imaging distortion correction method provided by the embodiments of the present application can be applied to image processing equipment.
  • the image processing device may be a camera, or may be a display device, or may be a video server connected to the display device.
  • the display device has a built-in camera, or the display device is connected to an external camera.
  • the camera is a wide-angle camera.
  • the camera is used to photograph the shooting area to collect images.
  • Image processing equipment is used to perform distortion correction on human body imaging in images collected by the camera.
  • the image processing device can also splice multiple distortion-corrected human body images so that the display device can display a spliced image containing multiple human body images.
  • the video server can be a server, a server cluster composed of multiple servers, or a cloud computing platform.
  • the display device may be a conference terminal, such as a large screen, an electronic whiteboard, a mobile phone, a tablet, or a smart wearable device or other electronic device with a display function.
  • the display device can be a smart TV, a projection device, or a virtual reality (VR) device, etc.
  • FIG. 6 is a schematic diagram of an application scenario involving an imaging distortion correction method provided by an embodiment of the present application.
  • the application scenario is a video conferencing scenario.
  • this application scenario includes multiple conference terminals 601A-601C (collectively referred to as conference terminals 601). Multiple conference terminals 601 are connected for communication.
  • the conference terminal 601A and the conference terminal 601B respectively have a built-in wide-angle camera (not shown in the figure), and the conference terminal 601C has a built-in non-wide-angle camera (not shown in the figure).
  • This application scenario also includes a video server 602, and multiple conference terminals 601 are connected to the video server 602 respectively. Communication connections are realized between multiple conference terminals 601 through a video server 602.
  • the video server 602 may be, for example, a multi-point control unit (MCU).
  • MCU multi-point control unit
  • FIG. 7 is a schematic diagram of the regional distribution of images collected by a wide-angle camera provided in an embodiment of the present application. As shown in Figure 7, the image includes distorted areas and non-distorted areas. The non-distorted area may be the center area of the image. The distortion area can be other areas in the image except the central area.
  • the conference terminal 601A or the video server 602 can perform distortion correction on the human body imaging in the images collected by the built-in wide-angle camera in the conference terminal 601A.
  • the conference terminal 601A or the video server 602 can also receive the image sent by the conference terminal 601B and collected by the built-in wide-angle camera of the conference terminal 601B, and perform distortion correction on the human body imaging in the image.
  • the conference terminal 601A or the video server 602 can also splice multiple human body images from one or more images.
  • the multiple human body images include human body images that are distorted but have been corrected and/or human body images that are not distorted.
  • the conference terminal 601A or the video server 602 can stitch multiple human body images in one image from one conference terminal.
  • the conference terminal 601A or the video server 602 may also splice multiple human body images in multiple images from multiple conference terminals.
  • the functions of the conference terminal 601B and the functions of the conference terminal 601C may refer to the functions of the conference terminal 601A, which will not be described in detail in the embodiment of this application.
  • the imaging distortion correction method provided by the embodiment of the present application can be used to correct the distortion of human body imaging in images collected by a wide-angle camera.
  • the imaging distortion correction method provided by the embodiments of the present application can also be used to correct the distortion of other foreground images in the images collected by the wide-angle camera, such as the imaging of cats, dogs, etc.
  • the embodiments of the present application do not limit the application scenarios. .
  • distortion correction of human body imaging is taken as an example for explanation.
  • FIG. 8 is a schematic flowchart of an imaging distortion correction method provided by an embodiment of the present application. This method can be applied to image processing equipment.
  • the image processing device may be, for example, any conference terminal 601 in the application scenario as shown in Figure 6 or a video server 602 connected to the conference terminal 601. As shown in Figure 8, the method includes:
  • Step 801 The image processing device acquires the first image collected by the first camera.
  • the first camera is a wide-angle camera.
  • the image collected by the first camera includes distorted areas and non-distorted areas.
  • the first image includes a first human body image located in a distortion area of the first image.
  • the distortion type of human body imaging located in the distortion area of the first image includes human body tilt and/or human body deformation.
  • the first image is collected by the first camera using linear projection, and the distortion type of the human body image located in the distortion area of the first image includes human body deformation.
  • the first image may be the image A shown in FIG. 1
  • the human body image located in the distortion area of the first image may refer to the human body image A1.
  • the distortion type of human body imaging located in the distortion area of the first image also includes human body tilt.
  • the first image may be the image B shown in FIG. 2
  • the human body image located in the distortion area of the first image may refer to the human body image B1.
  • the first image is collected by the first camera using Panini projection
  • the distortion types of the human body image located in the distortion area of the first image include human body tilt and human body deformation.
  • the first image may be the image C shown in FIG. 3
  • the human body imaging located in the distortion area of the first image may refer to the human body imaging C1.
  • the first image is collected by the first camera using spherical projection, and the distortion type of the human body image located in the distortion area of the first image includes human body tilt.
  • the first image may be the image D shown in FIG. 4
  • the human body imaging located in the distortion area of the first image may refer to the human body imaging D1.
  • the first camera may be a camera built into or connected to the image processing device, and the projection method used by the first camera is pre-stored in the image processing device.
  • the image processing device acquires the first image.
  • the image processing device may acquire the first image through the first camera.
  • the first image is the image collected by the local end.
  • the image processing device is the conference terminal 601A in the application scenario as shown in Figure 6.
  • the first camera is the built-in wide-angle camera of the conference terminal 601A.
  • the first image is collected by the built-in wide-angle camera of the conference terminal 601A and then transmitted to the conference terminal. 601A processor.
  • the image processing device may acquire the first image, or the image processing device may receive the first image sent by other devices.
  • the first camera may be a camera built into or connected to other devices, and the other devices also need to send the first image to the image processing device.
  • the first image is an image collected at the remote end.
  • the image processing device is the conference terminal 601A in the application scenario as shown in Figure 6.
  • the first camera is the wide-angle camera built into the conference terminal 601B. After the first image is collected by the wide-angle camera built into the conference terminal 601B, the first image is captured by the conference terminal 601B. 601B is sent to the conference terminal 601A.
  • Step 802 The image processing device performs distortion correction on the first human body image according to the first imaging position of the first human body image in the first image to obtain a second human body image.
  • the second human body imaging satisfies the following conditions: the tilt angle of the human body is less than a preset angle, and the deformation degree of the human body is less than a preset threshold.
  • the inclination angle of the human body may refer to the angle between the height direction of the human body and the vertical direction.
  • the image processing device can determine the imaging positions of all human body images in the first image, and then determine which human body image or images are located in the distortion area of the first image based on the imaging positions of the human body images.
  • the first human body image is any human body image located in the distortion area of the first image.
  • an implementation method for the image processing device to determine the imaging position of the human body image in the first image includes: the image processing device uses a face detection algorithm to determine the imaging position of the human face image in the first image, and then determines the imaging position of the human face image in the first image according to the human face The imaging position of the image determines the imaging position in the first image of the human body image to which the face image belongs.
  • the image processing device may, after acquiring the imaging position of the face imaging in the first image, expand the face imaging area in the first image to obtain an imaging area including human body imaging.
  • the human body imaging can be the upper body imaging of the human body, or it can be the whole body imaging of the human body, or it can also be the head and shoulder imaging of the human body.
  • FIG. 9 is a schematic diagram of an image provided by an embodiment of the present application.
  • the image includes user P's human body imaging and user Q's human body imaging.
  • the image processing device can expand the face imaging area P1 of user P in the image to obtain the human body imaging area P2 of user P in the image.
  • the image processing device can expand user Q's face imaging area Q1 in the image to obtain user Q's human body imaging area Q2 in the image.
  • the human body imaging area P2 and the human body imaging area Q2 both include upper body imaging of the human body.
  • the image processing device may also use other implementation methods to determine the imaging position of the human body in the first image.
  • the image processing device can segment the human body instance on the first image to obtain the human body mask in the first image, and then determine the human body mask corresponding to the imaging area in the first image, and the imaging area is the imaging position of the human body imaging.
  • the embodiments of the present application do not limit the implementation of determining the imaging position of human body imaging in the image.
  • the distortion type and/or degree of distortion of human body imaging in the image collected by the first camera is different.
  • the following embodiments of the present application are aimed at the case where the first camera adopts three different projection modes. Explain separately.
  • the first image is collected by the first camera using linear projection, and the distortion type of human body imaging located in the distortion area of the first image includes human body deformation.
  • step 802 includes the following steps 8021A to 8023A.
  • step 8021A the image processing device performs deformation correction projection transformation on the first image according to the field of view angle of the first camera and the focal length of the first camera to obtain a second image.
  • the deformation correction projection transformation is Panini projection transformation or spherical projection transformation.
  • the image processing device performs spherical projection transformation on the image A shown in Figure 1, and can obtain the image D shown in Figure 4, where the human body image D2 corresponds to the human body image A2, and the human body image D1 corresponds to the human body image A1.
  • human body imaging D1 has significantly improved human body shape, but the human body is tilted.
  • the image processing device needs to obtain the field of view angle of the first camera and the focal length of the first camera.
  • the field of view angle of the first camera and the focal length of the first camera can be pre-stored in the image processing device.
  • the field of view angle of the first camera and the focal length of the first camera may be sent to the image processing device by the remote end.
  • the image processing device is a first conference terminal, and the first image is collected by a first camera built into or externally connected to the second conference terminal and sent to the first conference terminal.
  • the second conference terminal can send the first image to the first conference terminal through the network.
  • the field of view angle of the first camera and the focal length of the first camera are sent to the first conference terminal.
  • the first conference terminal stores the camera field of view and camera focal length corresponding to the second conference terminal, so that after subsequently receiving an image from the second conference terminal, the image can be processed using the corresponding camera field of view and camera focal length.
  • the image processing device can estimate the field of view of the first camera used to collect the first image based on the first image, and then based on the estimate From the obtained field of view, the focal length of the first camera is estimated.
  • the image processing device can use a linear regression algorithm to predict the camera field of view corresponding to the image based on AlexNet (a deep convolutional neural network). This embodiment of the present application specifically predicts the camera field of view corresponding to the image. The process will not be described in detail.
  • step 8022A the image processing device determines the first human body image in the second image according to the first imaging position. Two imaging positions.
  • an implementation of step 8022A includes: the image processing device determines, according to the first imaging position, the field of view angle of the first camera, and the focal length of the first camera, that the first human body is imaged in the second image in the second image.
  • Imaging position The first imaging position can be represented by image coordinates under linear projection
  • the second imaging position can be represented by image coordinates under spherical projection.
  • the image processing device may convert the image coordinates under linear projection into image coordinates under spherical projection according to the field of view angle of the first camera and the focal length of the first camera.
  • FIG. 10 is a schematic diagram of image coordinate transformation provided by an embodiment of the present application.
  • the z-axis represents the direction of the main optical axis of the first camera
  • the x-axis represents the width direction of the image coordinate system
  • o represents the optical center of the first camera.
  • the angle between the line connecting the object M to the optical center o and the main optical axis of the first camera is ⁇
  • the maximum value of ⁇ is half of the field of view of the first camera
  • the focal length of the first camera is f
  • the straight-line projection of object M on the image plane yields M'
  • the abscissa of M' is r u .
  • r u f*tan ⁇ .
  • the projection of the intersection point N between the line connecting MM' and the sphere with o as the center and f as the radius on the image plane is the abscissa r d of M' after the spherical projection conversion.
  • the image processing device may first perform deformation correction projection transformation on the first image to obtain the second image, and then determine the imaging positions of all human body images in the second image.
  • the method of determining the imaging position of human body imaging in the second image may refer to the above-mentioned method of determining the imaging position of human body imaging in the first image, which will not be described again in the embodiment of the present application.
  • step 8023A the image processing device performs rotation correction on the first human body image cropped from the second image according to the second imaging position to obtain a second human body image.
  • the second human body image is perpendicular to the ground, that is, the second human body image is a vertical human body image.
  • the second imaging position includes the center point position of the human face and the center point position of the human body.
  • An implementation of step 8022A includes: the image processing device determines the human body tilt angle of the first human body imaged in the second image based on the position of the face center point and the human body center point position of the first human body imaged in the second image.
  • the image processing device performs rotation correction on the first human body image cropped from the second image according to the tilt angle of the first human body image in the second image.
  • the human body tilt angle can be defined as the angle of the line connecting the center point of the human face to the center point of the human body relative to the vertical line.
  • the image processing device can perform face detection and human body detection on the first human body image in the first image, obtain the face center point coordinates and human body center point coordinates of the first human body image in the first image, and then calculate The coordinates of the face center point and the human body center point in the second image after the first human body image undergoes deformation correction projection transformation.
  • the image processing device can perform face detection and human body detection on the first human body image in the second image, and directly obtain the face center point coordinates and human body center point coordinates of the first human body image in the second image.
  • the coordinates of the center point of the face of the first human body in the second image are (m1, n1), and the coordinates of the center point of the human face of the first human body in the second image are (m2, n2), then the coordinates of the center point of the first human body in the second image are (m2, n2).
  • the tilt angle of the human body in the second image is equal to arctan((m1-m2)/(n1-n2)). In the corresponding image with a camera field of view of 120°, the tilt angle of the human body imaged with the face located on the most edge diagonal is approximately 10 degrees.
  • the image processing device can first crop the second image to obtain a larger area containing the first human body image.
  • the greater the human body tilt angle The larger the area that needs to be cropped.
  • a smaller area containing the first human body image is then cropped.
  • the twice-cropped area containing the first human body image may be a rectangular area, or may be an area of other specified shapes.
  • the embodiment of the present application does not limit the shape of the cropping area.
  • FIG. 11 is a schematic diagram of rotation correction of human body imaging in the second image provided by an embodiment of the present application.
  • the image processing device first crops a rectangular area Z1 containing an oblique human body image from the second image, and then performs rotation correction on the rectangular area Z1 containing an oblique human body image, to obtain a rectangular area Z1 containing a vertical human body image.
  • the rectangular area Z2 containing the vertical human body image is cropped from the rectangular area Z1 containing the vertical human body image.
  • the range of the rectangular area Z1 can be determined based on the range of the rectangular area Z2 that ultimately needs to be cropped.
  • the image processing device can calculate the four vertices of the circumscribed rectangle of the rectangular area Z2 relative to the rectangular area Z2 The coordinates of the center point, and after performing affine transformation on the four vertex coordinates (rotating 10 degrees counterclockwise), the minimum range of the rectangular area Z1 is obtained.
  • the rectangular area Z1 is also the circumscribing rectangle of the rectangular area Z2.
  • the image processing device first performs deformation correction projection transformation on the image obtained by linear projection to improve the deformation of the human body in the image.
  • the human body imaging in the image after deformation correction projection transformation will be tilted.
  • the image processing device cuts out the tilted human body image from the image that has undergone the deformation correction projection transformation, and performs rotation correction on the tilted human body image, so that the tilt distortion of the human body imaging is also improved, and finally the result is that there is basically no distortion. human body imaging.
  • step 802 includes the following steps 8021B to 8022B.
  • step 8021B the image processing device determines the degree of human body deformation of the first human body imaged in the first image according to the first imaging position based on the pre-stored first correspondence relationship between the imaging position and the human body deformation degree.
  • the first correspondence relationship matches the field of view angle of the first camera. If the field of view of the wide-angle camera is different, the degree of deformation of the human body imaged in the distortion area in the image collected by the wide-angle camera using linear projection will be different. Generally speaking, the larger the field of view of a wide-angle camera, the more severe the deformation of the foreground image located in the distortion area of the image collected by the wide-angle camera.
  • the first correspondence that matches the field of view of the first camera refers to the correspondence between the imaging position and the deformation degree of the human body in the image collected under the field of view of the first camera.
  • the human body image in the image collected by the first camera using linear projection will be stretched in the horizontal direction.
  • the degree of stretching is determined by where the human body is imaged in the image. The closer the human body image is to the edge of the image, the more severe the stretching.
  • the width of the image is w
  • the distance from the position of the face in the image to the center of the image is p
  • the stretching degree of the face at the edge of the image is ⁇ , using linear interpolation.
  • the stretching degree ⁇ of the face at the edge of the image is determined by the camera's field of view.
  • the possible value of ⁇ is 1.2.
  • the corresponding relationship between the imaging position that matches the camera's field of view and the degree of human body deformation can be expressed using this linear interpolation calculation formula.
  • step 8022B the image processing device performs deformation correction on the first human body image cropped from the first image according to the degree of human deformation of the first human body image in the first image.
  • the image processing device may pre-store one or more sets of correspondences between imaging positions and human body deformation degrees, and each set of correspondences matches a camera field of view. In this way, the image processing device can determine the degree of human deformation of the human body image in the image based on the corresponding relationship between the imaging position matching the camera field of view corresponding to the image and the degree of human body deformation. After cropping the human body image from the image, deformation correction is performed on the cropped human body image, which can improve the human body deformation of the human body image. In addition, this implementation does not require deformation correction projection transformation, has a small amount of calculation, and has low processing performance requirements for image processing equipment.
  • the distortion type of human body imaging located in the distortion area of the first image also includes human body tilt. That is to say, in this case, the distortion type of the first human body image in the first image collected using the linear projection method includes human body deformation and human body tilt.
  • the human body deformation in human body imaging can be improved through deformation correction projection transformation (step 8021A), and the human body tilt in human body imaging can be improved through rotation correction (step 8023A), so
  • the image processing device does not need to deal with the problem caused by the tilt of the camera relative to the horizontal plane. Trapezoidal deformation problem.
  • step 802 in addition to performing deformation correction on the first human body image in the first image, it is also necessary to perform rotation correction on the first human body image in the first image.
  • embodiments of the present application provide the following three solutions for rotation correction of human body imaging.
  • the image processing device can first perform rotation correction on the human body image, and then perform deformation correction on the rotationally corrected human body image.
  • the image processing equipment first performs deformation correction on the human body image, and then performs rotation correction on the deformation-corrected human body image.
  • the first imaging position includes the center point position of the human face and the center point position of the human body.
  • the image processing device determines the human body tilt angle of the first human body imaged in the first image based on the position of the face center point of the first human body imaged in the first image and the human body center point position.
  • the image processing device performs rotation correction on the first human body image cropped from the first image according to the tilt angle of the first human body image in the first image.
  • the human body tilt angle can be defined as the angle of the line connecting the center point of the human face to the center point of the human body relative to the vertical line.
  • the image processing device determines the human body tilt angle at which the first human body is imaged in the first image based on the pre-stored second correspondence between the imaging position and the human body tilt angle.
  • the image processing device performs rotation correction on the first human body image cropped from the first image according to the tilt angle of the first human body image in the first image.
  • the second corresponding relationship matches the field of view angle of the first camera and the pitch angle of the first camera. If the field of view of the wide-angle camera is different, and/or the pitch angle of the wide-angle camera is different, the tilt angle of the human body located in the distortion area in the image collected by the wide-angle camera using linear projection will be different. Generally speaking, the larger the field of view and pitch angle of the wide-angle camera, the larger the tilt angle of the foreground image located in the distortion area of the image collected by the wide-angle camera.
  • the second corresponding relationship that matches the field of view angle of the first camera and the pitch angle of the first camera refers to the relationship between the imaging position and the tilt angle of the human body in the image collected under the field of view angle and pitch angle of the first camera.
  • the pitch angle of the camera is 5 degrees, that is, the camera is tilted 5 degrees downward compared to the horizontal plane, then in the image collected by the camera, the human body at the edge of the image will be the image of the human body.
  • the tilt angle is about 10 degrees. If the pitch angle of the camera is 15 degrees, that is, the camera is tilted 15 degrees downward compared to the horizontal plane, then in the image collected by the camera, the tilt angle of the human body imaged at the edge of the image is approximately 20 degrees.
  • the pitch angle of the camera can be manually input into the image processing device after the camera is deployed, or it can also be measured or calculated by the device itself using hardware or software.
  • the hardware can integrate sensors that can determine the device's placement angle on the camera or the device where the camera is deployed, such as an acceleration sensor or a gravity sensor.
  • the software can use the image processing device to calculate the inclination of the vertical lines in the shooting scene based on the images captured by the camera.
  • the vertical lines in the shooting scene can be, for example, the left and right edge lines of windows, TVs or projection screens.
  • the image processing device can first perform target detection, classification and segmentation on the image, then determine the tilt direction of the camera based on the segmentation results, and extract the edge line of the target based on the segmentation results to calculate the tilt angle of the edge line relative to the vertical line. Then get the pitch angle of the camera.
  • the embodiment of the present application does not limit the way in which the image processing device obtains the camera pitch angle.
  • one or more sets of correspondences between imaging positions and human body tilt angles can be pre-stored in the image processing device, and each set of correspondences matches a camera field of view angle and a camera pitch angle.
  • the image processing device can determine the position of the human body in the image based on the corresponding relationship between the imaging position matching the camera field of view and the camera pitch angle corresponding to the image and the tilt angle of the human body in the image.
  • Human body tilt angle after cropping the human body image from the image, first perform rotation correction on the cropped human body image, and then further perform deformation correction on the human body image obtained through rotation correction, which can improve the human body tilt and human body deformation in the human body imaging.
  • this solution does not need to calculate the human body tilt angle for human body imaging, has a small amount of calculation, and has low processing performance requirements for image processing equipment.
  • the image processing device needs to first crop the human body image from the original image (such as the first image) collected by the camera, and then perform rotation correction and deformation correction on the cropped human body image.
  • the relevant description in the above step 8023A which will not be described again in this embodiment of the present application.
  • the image processing device uses a trapezoidal transformation matrix corresponding to the pitch angle of the first camera to perform trapezoidal transformation on the first image to obtain a third image. Then the image processing device performs distortion correction on the first human body image according to the third imaging position of the first human body image in the third image.
  • the trapezoidal transformation is a perspective transformation used to establish the correspondence between two planar fields. Perform a trapezoidal transformation on an image, that is, project the image from one viewing plane to another. Once the pitch angle of the camera is known, the degree to which the image needs to be corrected is known. By inputting to the image processing device the camera coordinates corresponding to multiple points (greater than or equal to 4 points) in the original image and the camera coordinates corresponding to the multiple points in the target image that is desired to be corrected, the image processing device can solve and obtain the trapezoid. transformation matrix.
  • the camera coordinates refer to the three-dimensional coordinates in the camera coordinate system.
  • the camera coordinate system is a three-dimensional rectangular coordinate system established with the optical center of the camera as the origin and the main optical axis as the z-axis.
  • the x-axis of the camera coordinate system is parallel to the x-axis of the image coordinate system corresponding to the image collected by the camera.
  • the y-axis of the camera coordinate system is parallel to the y-axis of the image coordinate system corresponding to the image collected by the camera.
  • the image processing device can obtain a 3*3 trapezoidal transformation matrix H based on the corresponding camera coordinates of multiple points in the original image and the corresponding camera coordinates in the target image based on the following formula.
  • Target image original image*H.
  • this solution since the image collected by the camera will undergo trapezoidal deformation when the camera is tilted relative to the horizontal plane, this solution first performs trapezoidal transformation on the image collected by the camera to improve the trapezoidal deformation of the image, and then further transforms the image. Deformation correction of the human body image in the image can improve the display effect of the final human body image.
  • the above third solution can also be used in conjunction with the first implementation of step 802 in the above first possible situation, that is, before step 8021A is executed, the image processing device first performs trapezoidal transformation on the first image to obtain the third image. , the subsequent image processing device performs further image processing on the third image.
  • the above-mentioned image processing device performs distortion correction on the first human body image according to the third imaging position in the third image, which may include: the image processing device performs distortion correction on the first human body image in the third image according to the third imaging position.
  • the cropped first human body image is scaled horizontally.
  • the first image is collected by the first camera using Panini projection
  • the distortion types of human body imaging located in the distortion area of the first image include human body tilt and human body deformation.
  • One implementation of step 802 includes: the image processing device performs spherical projection transformation on the first image according to the field of view angle of the first camera and the focal length of the first camera to obtain a fourth image.
  • the image processing device determines a fourth imaging position in the fourth image where the first human body is imaged based on the first imaging position.
  • the image processing device rotates and corrects the first human body image cropped from the fourth image according to the fourth imaging position to obtain a second human body image.
  • the image processing device can first perform linear projection transformation on the first image collected using Panini projection according to the field of view angle of the first camera and the focal length of the first camera, and then perform linear projection transformation on the image that has undergone linear projection transformation. Spherical projection transformation is performed to obtain the fourth image.
  • the image processing device performs rotational correction on the first human body image cropped from the fourth image according to the fourth imaging position. Reference may be made to the above step 8023A.
  • the image processing device performs rotation correction on the second image according to the second imaging position. The implementation method of performing rotation correction on the cropped first human body image will not be described in detail here in the embodiments of the present application.
  • the first image is collected by the first camera using a spherical projection method, and the distortion type of the human body image located in the distortion area of the first image includes human body tilt.
  • One implementation of step 802 includes: the image processing device performs rotation correction on the first human body image cropped from the first image according to the first imaging position.
  • the image processing device performs rotation correction on the first human body image cropped from the first image according to the first imaging position.
  • the image processing device can use corresponding strategies to improve the human body deformation and/or human body tilt problems in the human body imaging in the image according to the different projection methods used by the wide-angle camera.
  • Distortion-corrected human body imaging can be used in a variety of display mode scenarios that require splicing display.
  • the image processing device may also perform the following steps 803 to 805.
  • Step 803 The image processing device acquires the third human body image.
  • the third human body imaging is a human body image cropped from the non-distorted area of the target image, or the third human body imaging is a human body image located in the distortion area of the target image and has undergone distortion correction.
  • the target image is the first image, that is, the first human body imaging and the third human body imaging come from the same image.
  • the first human body image is the human body image A1 in the image A as shown in FIG. 1
  • the third human body image is the human body image A2 in the image A as shown in FIG. 1 (the human body image without distortion).
  • the target image is an image other than the first image, that is, the first human body imaging and the third human body imaging come from different images.
  • the first human body image is the human body image A1 in image A as shown in FIG. 1
  • the third human body image is the human body image C1 in image C as shown in FIG. 3 after distortion correction.
  • the target image is an image collected by a second camera, and the second camera is different from the first camera. That is, the first human body image and the third human body image are respectively from images collected by different cameras.
  • the second camera collects the target image at the same time as the first camera collects the first image.
  • Step 804 The image processing device splices the second human body image and the third human body image to obtain a spliced image.
  • the second human body imaging is obtained based on the first human body imaging from the first image through distortion correction.
  • the third human body image is a human body image cropped from the non-distorted area of the first image, or the third human body image is located in the distortion area of the first image and has undergone distortion correction.
  • the image processing device splices the second human body image and the third human body image, that is, splicing different human body images from the same image. Since the human body image used for splicing has undergone distortion correction or has no distortion in the original image, Therefore, after splicing multiple human body images together, the human body deformation and human body tilt degree of different human body images in the spliced picture can be basically consistent, and the coordination of the spliced picture is better, thereby ensuring the display effect.
  • the third human body image is a human body image cropped from a non-distorted area of an image other than the first image, or the third human body image is a human body image located in a non-distorted area other than the first image.
  • the distortion area of the image and the human body imaging after distortion correction.
  • the image processing device splices the second human body image and the third human body image, that is, splices the human body images from different images.
  • multiple conference terminals collect video conference images at the same time.
  • the image processing equipment can splice human body imaging from different video conference images. By putting people in different conference rooms into the same spliced picture, It can make multiple participants feel like they are working together, thereby improving the multi-person collaboration atmosphere.
  • the image processing device in response to receiving the selection instruction for the target display mode, the image processing device begins to perform the above step 802.
  • the target display mode can be smart equalization mode or gallery mode.
  • the image processing device performs a zoom operation on the second human body imaging and/or the third human body imaging according to the target display mode, and splices the second human body imaging and the third human body imaging that have undergone the zoom operation to obtain a spliced image.
  • the image processing device can intercept a rectangular area containing human body imaging, and make the height of each rectangular area equal to the height of the spliced screen through a scaling operation.
  • the image processing device can uniformly intercept the whole body imaging or the upper body imaging of the human body.
  • FIG. 12 is a schematic diagram of a spliced image in an intelligent equalization mode provided by an embodiment of the present application. As shown in Figure 12, the stitched image consists of four rectangular areas containing full-body imaging of the human body. It is obtained by splicing, and the heights of the rectangular areas where different human body images are located are the same.
  • the image processing device can intercept a rectangular area containing the human body image. Compared with the intelligent equalization mode, the image processing device can intercept the human body image in a smaller range in the gallery mode. For example, the image processing device can intercept the upper body image of the human body. and/or head and shoulders imaging.
  • FIGs 13 to 15 are respectively schematic diagrams of spliced images in gallery mode provided by embodiments of the present application. As shown in Figure 13, the spliced image is obtained by splicing four rectangular areas containing the head and shoulder images of the human body in the form of a four-square grid.
  • the spliced image is obtained by splicing nine rectangular areas containing the head and shoulder images of the human body in the form of a nine-square grid.
  • the spliced image is obtained by splicing 5 rectangular areas containing head and shoulder images of the human body, of which 4 human bodies are imaged as small images and 1 human body is imaged as a large image.
  • Step 805 The image processing device outputs the spliced image for display on the screen.
  • the image processing device outputs the spliced image for display on the screen.
  • the spliced image may be displayed on its own screen, or the spliced image may be sent to other devices for display by other devices.
  • the conference terminal can splice and display the human body images of multiple participants based on the images collected by the local end. In this way, participants with scattered seats can be displayed together in the spliced screen.
  • the conference terminal can also send spliced images to other remote conference terminals for display by other conference terminals.
  • the image processing device performs distortion correction on the human body image according to the imaging position of the human body image in the image. Since the degree of distortion of the human body imaging in the image is affected by the imaging position, the closer the imaging position is to the edge of the image, the more serious the degree of distortion of the human body imaging.
  • the human body images at different degrees of distortion of the human body images at different positions in the image can be spliced together to achieve different human bodies in the spliced picture.
  • the imaged human body deformation and human body tilt are consistent, which improves the coordination of the spliced images and thus improves the display effect.
  • the sequence of the steps of the imaging distortion correction method provided by the embodiments of the present application can be adjusted appropriately, and the steps can also be increased or decreased accordingly according to the situation. Any person familiar with the technical field can easily think of modified methods within the technical scope disclosed in this application, and they should be covered by the protection scope of this application.
  • the image processing device can send the obtained distortion-corrected human body image to other devices for splicing display by other devices.
  • the embodiments of this application will not be described in detail one by one.
  • FIG. 16 is a schematic structural diagram of an imaging distortion correction device provided by an embodiment of the present application.
  • the imaging distortion correction device 1600 includes: a first acquisition module 1601 and a distortion correction module 1602.
  • the first acquisition module 1601 is used to acquire the first image collected by the first camera.
  • the first image includes the first human body image located in the distortion area of the first image.
  • the distortion type of the human body imaging located in the distortion area of the first image includes the human body. Tilt and/or body deformation.
  • the distortion correction module 1602 is configured to perform distortion correction on the first human body image according to the first imaging position of the first human body image in the first image to obtain a second human body image.
  • the first image is collected by the first camera using linear projection, and the distortion type of human body imaging located in the distortion area of the first image includes human body deformation.
  • the distortion correction module 1602 is configured to perform a distortion correction projection transformation on the first image according to the field of view angle of the first camera and the focal length of the first camera to obtain a second image.
  • the second imaging position of the first human body image in the second image is determined according to the first imaging position. Rotate the first human body image cropped from the second image according to the second imaging position. Turn to correction.
  • the distortion correction module 1602 is configured to determine the second imaging position of the first human body in the second image based on the first imaging position, the field of view angle of the first camera, and the focal length of the first camera.
  • the second imaging position includes the position of the center point of the human face and the position of the center point of the human body.
  • the distortion correction module 1602 is configured to: determine whether the first human body is imaged in the second image according to the position of the center point of the human face and the position of the human body center point. human body tilt angle. Rotation correction is performed on the first human body image cropped from the second image according to the human body tilt angle of the first human body image in the second image.
  • the distortion correction module 1602 is configured to: based on the pre-stored first correspondence relationship between the imaging position and the degree of human body deformation, determine the degree of human body deformation of the first human body in the first image according to the first imaging position, wherein, The first correspondence matches the field of view of the first camera. According to the degree of human deformation of the first human body image in the first image, deformation correction is performed on the first human body image cropped from the first image.
  • the first camera is tilted relative to the horizontal plane
  • the distortion type of human body imaging located in the distortion area of the first image also includes human body tilt
  • the first imaging position includes the position of the center point of the human face and the position of the center point of the human body.
  • the distortion correction module 1602 is used to: determine the human body tilt angle of the first human body in the first image based on the position of the center point of the human face and the position of the human body center point. Rotation correction is performed on the first human body image cropped from the first image according to the human body tilt angle of the first human body image in the first image.
  • the first camera is tilted relative to the horizontal plane, and the distortion type of human body imaging located in the distortion area of the first image also includes human body tilt.
  • the distortion correction module 1602 is also configured to: use a method corresponding to the pitch angle of the first camera.
  • the trapezoidal transformation matrix performs trapezoidal transformation on the first image to obtain the third image. Distortion correction is performed on the first human body image according to the third imaging position of the first human body image in the third image.
  • the distortion correction module 1602 is configured to horizontally scale the first human body image cropped from the third image according to the third imaging position.
  • the first image is collected by the first camera using Panini projection
  • the distortion types of human body imaging located in the distortion area of the first image include human body tilt and human body deformation.
  • the distortion correction module 1602 is configured to perform spherical projection transformation on the first image according to the field of view angle of the first camera and the focal length of the first camera to obtain a fourth image.
  • a fourth imaging position of the first human body in the fourth image is determined according to the first imaging position. Rotation correction is performed on the first human body image cropped from the fourth image according to the fourth imaging position.
  • the first image is collected by the first camera using spherical projection, and the distortion type of human body imaging located in the distortion area of the first image includes human body tilt.
  • the distortion correction module 1602 is configured to perform rotation correction on the first human body image cropped from the first image according to the first imaging position.
  • the imaging distortion correction device 1600 also includes: a second acquisition module 1603, a splicing module 1604 and an output module 1605.
  • the second acquisition module 1603 is used to acquire a third human body image.
  • the third human body image is a human body image cropped from the non-distorted area of the target image, or the third human body image is located in the distortion area of the target image and has undergone distortion.
  • Corrected human imaging The splicing module 1604 is used to splice the second human body imaging and the third human body imaging to obtain a spliced image.
  • the output module 1605 is used to output the spliced image for display on the screen.
  • the target image is a first image, or the target image is an image collected by a second camera, and the second camera is different from the first camera.
  • the distortion correction module 1602 is configured to perform distortion correction on the first human body image according to the first imaging position in response to receiving the selection instruction for the target display mode.
  • the splicing module 1604 is configured to perform a scaling operation on the second human body imaging and/or the third human body imaging according to the target display mode, and splice the second human body imaging and the third human body imaging after the scaling operation to obtain a spliced image.
  • FIG. 18 is a schematic diagram of the hardware structure of an imaging distortion correction device provided by an embodiment of the present application.
  • the imaging distortion correction device 1800 includes a processor 1801 and a memory 1802 .
  • the memory 1801 and the memory 1802 are connected through a bus 1803 .
  • Figure 18 illustrates the processor 1801 and the memory 1802 independently of each other.
  • processor 1801 and memory 1802 are integrated together.
  • the imaging distortion correction device 1800 in Figure 18 is any conference terminal 601 or video server 602 in the application scenario shown in Figure 6.
  • the memory 1802 is used to store computer programs, which include operating systems and program codes.
  • Memory 1802 is various types of storage media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory, EEPROM), compact disc read-only memory (CD-ROM), flash memory, optical memory, register, optical disk storage, optical disk storage, magnetic disk or other magnetic storage device.
  • ROM read-only memory
  • RAM random access memory
  • EEPROM electrically erasable programmable read-only memory
  • CD-ROM compact disc read-only memory
  • flash memory optical memory, register, optical disk storage, optical disk storage, magnetic disk or other magnetic storage device.
  • the processor 1801 is a general-purpose processor or a special-purpose processor.
  • Processor 1801 may be a single-core processor or a multi-core processor.
  • the processor 1801 includes at least one circuit to execute the above imaging distortion correction method provided by the embodiment of the present application.
  • the imaging distortion correction device 1800 also includes a network interface 1804, which is connected to the processor 1801 and the memory 1802 through a bus 1803.
  • the network interface 1804 enables the imaging distortion correction device 1800 to communicate with other devices.
  • the processor 1801 can communicate with other devices through the network interface 1804 to obtain images collected by the camera, etc.
  • the imaging distortion correction device 1800 also includes an input/output (I/O) interface 1805.
  • the I/O interface 1805 is connected to the processor 1801 and the memory 1802 through a bus 1803.
  • the processor 1801 can receive input commands or data through the I/O interface 1805.
  • the I/O interface 1805 is used to connect the imaging distortion correction device 1800 to input devices, such as a keyboard, a mouse, etc.
  • the above-mentioned network interface 1804 and I/O interface 1805 are collectively referred to as communication interfaces.
  • the imaging distortion correction device 1800 also includes a display 1806 , which is connected to the processor 1801 and the memory 1802 through a bus 1803 .
  • the display 1806 can be used to display intermediate results and/or final results generated by the processor 1801 when executing the above method, for example, displaying a spliced image.
  • the display 1806 is a touch display screen to provide a human-computer interaction interface.
  • the bus 1803 is any type of communication bus used to interconnect internal devices of the imaging distortion correction device 1800 .
  • system bus for example, system bus.
  • the embodiment of the present application takes the above-mentioned devices inside the imaging distortion correction device 1800 as an example to be interconnected through the bus 1803.
  • the above-mentioned devices inside the imaging distortion correction device 1800 communicate with each other using other connection methods besides the bus 1803.
  • the above-mentioned devices inside the imaging distortion correction device 1800 are interconnected through a logical interface inside the imaging distortion correction device 1800 .
  • the above-mentioned devices may be arranged on separate chips, or at least part or all of them may be arranged on the same chip. Whether each device is independently installed on different chips or integrated on one or more chips often depends on the needs of product design.
  • the embodiments of this application do not limit the specific implementation forms of the above devices.
  • the imaging distortion correction device 1800 shown in FIG. 18 is only exemplary. During the implementation process, the imaging distortion correction device 1800 includes other components, which will not be listed one by one in this article.
  • the imaging distortion correction device 1800 shown in FIG. 18 can realize distortion correction of human body imaging by executing all or part of the steps of the method provided in the above embodiment.
  • Embodiments of the present application also provide a computer-readable storage medium. Instructions are stored on the computer-readable storage medium. When the instructions are executed by a processor, the imaging distortion correction method shown in Figure 6 is implemented.
  • An embodiment of the present application also provides a computer program product, including a computer program.
  • the computer program is executed by a processor, the imaging distortion correction method shown in Figure 6 is implemented.
  • the information including but not limited to user equipment information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • signals involved in this application All are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
  • the image data involved in this application were obtained with full authorization.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

本申请公开了一种成像畸变矫正方法及装置，属于图像处理技术领域。图像处理设备获取第一相机采集的第一图像。第一图像包括位于第一图像的畸变区域的第一人体成像。位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和/或人体形变。图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正，得到第二人体成像。本申请通过针对位于图像不同位置的人体成像的不同畸变程度，分别对人体成像进行矫正，这样后续将位于图像的不同位置的、分别经过畸变矫正的人体成像拼接在一起之后，可以使得拼接画面中不同人体成像的人体形变和人体倾斜程度保持一致，提高了拼接画面的协调性，从而提高了显示效果。

Description

成像畸变矫正方法及装置 技术领域
本申请涉及图像处理技术领域，特别涉及一种成像畸变矫正(lens distortion correction，LDC)方法及装置。
背景技术
广角相机被越来越广泛地应用到视频会议场景中。视场角(field of view，FOV)大于60度的相机通常可以被称为广角相机。
在一些视频会议场景中，需要将广角相机采集的视频会议图像中的多个人体成像裁剪出来后拼接显示。然而在当前的技术中，将从图像的不同位置裁剪到的人体成像拼接在一起之后，会出现拼接画面不协调的情况，导致显示效果较差。
发明内容
本申请提供了一种成像畸变矫正方法及装置，可以解决目前基于多个人体成像得到的拼接画面不协调导致显示效果较差的问题。
第一方面，提供了一种成像畸变矫正方法。该方法应用于图像处理设备，该图像处理设备例如可以是会议终端或视频服务器等。该方法包括：图像处理设备获取第一相机采集的第一图像。第一图像包括位于第一图像的畸变区域的第一人体成像。位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和/或人体形变。图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正，得到第二人体成像。
其中，第二人体成像满足：人体倾斜角度小于预设角度，且人体形变程度小于预设阈值。本申请中提及的人体倾斜，是指人体高度方向(例如人脸中心点与人体中心点的连线)与竖直方向存在一定夹角，相应地，人体倾斜角度是指人体高度方向与竖直方向的夹角。
本申请中，由于发明人发现图像中的人体成像的畸变程度受成像位置的影响，成像位置越靠近图像边缘，则人体成像的畸变程度越严重。因而，通过针对位于图像不同位置的人体成像的不同畸变程度，分别对人体成像进行矫正，这样后续将位于图像的不同位置的、分别经过畸变矫正的人体成像拼接在一起之后，可以使得拼接画面中不同人体成像的人体形变和人体倾斜程度保持一致，提高了拼接画面的协调性，从而提高了显示效果。
第一种实现方式，第一图像由第一相机采用直线投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体形变。
结合第一种实现方式，图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正的实现过程，包括：图像处理设备根据第一相机的视场角和第一相机的焦距，对第一图像进行形变矫正投影变换，得到第二图像。图像处理设备根据第一成像位置确定第一人体成像在第二图像中的第二成像位置。图像处理设备根据第二成像位置，对从第二图像中裁剪得到的第一人体成像进行旋转矫正。
本申请中，图像处理设备可以先对采用直线投影方式得到的图像进行形变矫正投影变换，以改善图像中人体成像的人体形变，但经过形变矫正投影变换的图像中的人体成像会发生倾斜。然后图像处理设备再从经过形变矫正投影变换的图像中裁剪得到发生倾斜的人体成像，并对该发生倾斜的人体成像进行旋转矫正，使得人体成像的倾斜畸变也得到改善，最终得到 基本不存在畸变的人体成像。
可选地，图像处理设备根据第一成像位置确定第一人体成像在第二图像中的第二成像位置的一种实现方式，包括：图像处理设备根据第一成像位置、第一相机的视场角和第一相机的焦距，确定第一人体成像在第二图像中的第二成像位置。
可选地，第二成像位置包括人脸中心点位置和人体中心点位置。图像处理设备根据第二成像位置，对从第二图像中裁剪得到的第一人体成像进行旋转矫正的实现方式，包括：图像处理设备根据人脸中心点位置和人体中心点位置，确定第一人体成像在第二图像中的人体倾斜角度。图像处理设备根据第一人体成像在第二图像中的人体倾斜角度，对从第二图像中裁剪得到的第一人体成像进行旋转矫正。
可选地，形变矫正投影变换为帕尼尼投影变换或球面投影变换。
或者，结合第一种实现方式，图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正的实现过程，包括：图像处理设备基于预先存储的成像位置与人体形变程度的第一对应关系，根据第一成像位置确定第一人体成像在第一图像中的人体形变程度，其中，第一对应关系与第一相机的视场角相匹配。图像处理设备根据第一人体成像在第一图像中的人体形变程度，对从第一图像中裁剪得到的第一人体成像进行形变矫正。
本申请中，图像处理设备中可以预先存储有一组或多组成像位置与人体形变程度的对应关系，每组对应关系与一个相机视场角相匹配。这样，图像处理设备可以基于与图像对应的相机视场角相匹配的成像位置与人体形变程度的对应关系，根据人体成像在该图像中的成像位置确定人体成像在该图像中的人体形变程度，在从该图像中裁剪得到人体成像之后，再对裁剪得到的人体成像进行形变矫正，可以改善人体成像的人体形变。另外，这种实现方式无需进行形变矫正投影变换，计算量小，对图像处理设备的处理性能要求较低。
可选地，第一相机相对于水平面倾斜设置，位于第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜。第一成像位置包括人脸中心点位置和人体中心点位置，图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正的实现过程，还包括：图像处理设备根据人脸中心点位置和人体中心点位置，确定第一人体成像在第一图像中的人体倾斜角度。图像处理设备根据第一人体成像在第一图像中的人体倾斜角度，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。
或者，第一相机相对于水平面倾斜设置，位于第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜。图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正的实现过程，还包括：图像处理设备基于预先存储的成像位置与人体倾斜角度的第二对应关系，确定第一人体成像在第一图像中的人体倾斜角度，其中，第二对应关系与第一相机的视场角和第一相机的俯仰角相匹配。图像处理设备根据第一人体成像在第一图像中的人体倾斜角度，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。
这种方案中，图像处理设备预先存储有一组或多组成像位置与人体倾斜角度的对应关系，每组对应关系与一个相机视场角和一个相机俯仰角相匹配。这样，图像处理设备可以基于与图像对应的相机视场角和相机俯仰角相匹配的成像位置与人体倾斜角度的对应关系，根据人体成像在该图像中的成像位置确定人体成像在该图像中的人体倾斜角度，在从该图像中裁剪得到人体成像之后，先对裁剪得到的该人体成像进行旋转矫正，再进一步可以对经过旋转矫正得到人体成像进行形变矫正，可以改善人体成像的人体倾斜和人体形变。另外，本方案无需计算人体成像的人体倾斜角度，计算量小，对图像处理设备的处理性能要求较低。
又或者，第一相机相对于水平面倾斜设置，位于第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜。图像处理设备在获取第一相机采集的第一图像之后，图像处理设备根据第一相机的俯仰角对第一图像进行梯形变换，得到第三图像。相应地，图像根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正的实现方式，包括：图像处理设备根据第一人体成像在第三图像中的第三成像位置，对第一人体成像进行畸变矫正。
这种方案中，由于在相机相对于水平面倾斜设置的情况下，相机采集的图像会发生梯形形变，本方案通过对相机采集到的图像先进行梯形变换，以改善图像的梯形形变，进一步再对图像中的人体成像进行形变矫正，可以提高最终获取的人体成像的显示效果。
可选地，图像处理设备根据第一人体成像在第三图像中的第三成像位置，对第一人体成像进行畸变矫正的实现方式，包括：图像处理设备根据第三成像位置，对从第三图像中裁剪得到的第一人体成像进行水平缩放。
第二种实现方式，第一图像由第一相机采用帕尼尼投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和人体形变。图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正的实现过程，包括：图像处理设备根据第一相机的视场角和第一相机的焦距，对第一图像进行球面投影变换，得到第四图像。图像处理设备根据第一成像位置确定第一人体成像在第四图像中的第四成像位置。图像处理设备根据第四成像位置，对从第四图像中裁剪得到的第一人体成像进行旋转矫正。
第三种实现方式，第一图像由第一相机采用球面投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜。图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正的实现过程，包括：图像处理设备根据第一成像位置，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。
可选地，图像处理设备还可以获取第三人体成像，第三人体成像为从目标图像的非畸变区域中裁剪得到的人体成像，或者，第三人体成像为位于目标图像的畸变区域、且经过畸变矫正的人体成像。图像处理设备对第二人体成像和第三人体成像进行拼接，得到拼接图像。图像处理设备输出拼接图像到屏幕上显示。
本申请中，图像处理设备输出拼接图像到屏幕上显示，可以是自身屏幕上显示拼接图像，或者也可以是向其它设备发送拼接图像以供其它设备显示。例如视频会议场景中，会议终端可以基于本端采集的图像对多个与会人员的人体成像进行拼接显示，这样可以将座位比较分散的与会人员集中显示在拼接画面中。会议终端还可以向远端的其它会议终端发送拼接图像，以供其它会议终端显示。
可选地，上述目标图像为第一图像，或者，上述目标图像为第二相机采集的图像，第二相机与第一相机不同。也即是，第一人体成像和第三人体成像来自同一图像，或者，第一人体成像和第三人体成像分别来自不同相机采集的图像。
本申请中，图像处理设备可以对来自同一图像的不同人体成像进行拼接，由于用于拼接的人体成像经过畸变矫正或在原始图像中就未发生畸变，因此在将多个人体成像拼接在一起之后，可以使得拼接画面中不同人体成像的人体形变和人体倾斜程度基本保持一致，拼接画面的协调性较好，进而可以保证显示效果。或者，图像处理设备还可以对来自不同图像的人体成像进行拼接，例如在视频会议场景中，多个会议终端在同一时刻分别采集视频会议图像，图像处理设备可以对来自不同视频会议图像的人体成像进行拼接，通过将不同会议室的人放到同一个拼接画面中，可以让多个参会方感觉像是在一起工作，从而能够改善多人协作氛围。
可选地，图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成 像进行畸变矫正的一种实现方式，包括：响应于接收到对目标显示模式的选择指令，图像处理设备根据第一成像位置，对第一人体成像进行畸变矫正。相应地，图像处理设备对第二人体成像和第三人体成像进行拼接，得到拼接图像的实现方式，包括：按照目标显示模式对第二人体成像和/或第三人体成像执行缩放操作，并对经过缩放操作的第二人体成像和第三人体成像进行拼接，得到拼接图像。
其中，目标显示模式可以是智能均分模式或画廊模式。
可选地，图像处理设备为会议终端，第一相机内置在图像处理设备中，或者，第一相机与图像处理设备相连。这种情况下，第一图像为本端采集的图像。
可选地，图像处理设备获取第一相机采集的第一图像，包括：图像处理设备接收其它设备发送的第一图像。第一相机可以是其它设备内置或与其它设备相连的相机，其它设备还需向图像处理设备发送第一相机所采用的投影方式。这种情况下，第一图像为远端采集的图像。
这种实现方式下，第一相机的视场角和第一相机的焦距可以由其它设备发送给图像处理设备。或者，图像处理设备接收到第一图像之后，可以基于第一图像估算出用于采集第一图像的第一相机的视场角，再根据估算得到的视场角，估算出第一相机的焦距。例如，图像处理设备可以采用线性回归算法，基于AlexNet(一种深度卷积神经网络)对图像对应的相机视场角进行预测。
第二方面，提供了一种成像畸变矫正装置。所述装置包括多个功能模块，所述多个功能模块相互作用，实现上述第一方面及其各实施方式中的方法。所述多个功能模块可以基于软件、硬件或软件和硬件的结合实现，且所述多个功能模块可以基于具体实现进行任意组合或分割。
第三方面，提供了一种成像畸变矫正装置，包括：处理器和存储器；
所述存储器，用于存储计算机程序，所述计算机程序包括程序指令；
所述处理器，用于调用所述计算机程序，实现上述第一方面及其各实施方式中的方法。
第四方面，提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有指令，当所述指令被处理器执行时，实现上述第一方面及其各实施方式中的方法。
第五方面，提供了一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时，实现上述第一方面及其各实施方式中的方法。
第六方面，提供了一种芯片，芯片包括可编程逻辑电路和/或程序指令，当芯片运行时，实现上述第一方面及其各实施方式中的方法。
附图说明
图1是本申请实施例提供的一种由广角相机采用直线投影方式采集得到的图像示意图；
图2是本申请实施例提供的另一种由广角相机采用直线投影方式采集得到的图像示意图；
图3是本申请实施例提供的一种由广角相机采用球面投影方式采集得到的图像示意图；
图4是本申请实施例提供的一种由广角相机采用帕尼尼投影方式采集得到的图像示意 图；
图5是本申请实施例提供的一种人体成像的拼接画面示意图；
图6是本申请实施例提供的一种应用场景示意图；
图7是本申请实施例提供的一种广角相机采集的图像的区域分布示意图；
图8是本申请实施例提供的一种成像畸变矫正方法的流程示意图；
图9是本申请实施例提供的一种图像示意图；
图10是本申请实施例提供的一种图像坐标变换示意图；
图11是本申请实施例提供的一种对第二图像中的人体成像进行旋转矫正的示意图；
图12是本申请实施例提供的一种智能均分模式下的拼接图像示意图；
图13是本申请实施例提供的一种画廊模式下的拼接图像示意图；
图14是本申请实施例提供的另一种画廊模式下的拼接图像示意图；
图15是本申请实施例提供的又一种画廊模式下的拼接图像示意图；
图16是本申请实施例提供的一种成像畸变矫正装置的结构示意图；
图17是本申请实施例提供的另一种成像畸变矫正装置的结构示意图；
图18是本申请实施例提供的一种成像畸变矫正装置的硬件结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
目前的广角相机可以采用直线投影(rectilinear)方式、帕尼尼(pannini)投影方式或球面投影方式采集图像。其中，直线投影方式和帕尼尼投影方式是目前较为主流的投影方式。帕尼尼投影方式相较于直线投影方式，适合视场角更大的广角相机。
在一些视频会议场景中，需要将广角相机采集的视频会议图像中的多个人体成像裁剪出来后拼接显示。然而在当前的技术中，无论广角相机采用哪种投影方式，将从广角相机采集的图像中不同位置裁剪到的人体成像拼接在一起之后，都会出现拼接画面不协调的情况，导致显示效果较差。实际研究中，发明人发现，在广角相机采集的图像中，人体成像的畸变程度与人体在图像中的位置相关，通常越靠近图像边缘的人体成像的畸变程度越严重。
例如，对于采用直线投影方式的广角相机，在广角相机垂直于地面设置的情况下，即广角相机的主光轴平行于地面(即水平面)时，这种投影方式能够使广角相机拍摄到的图像中的水平线很直且平行于地面，竖直线也很直且垂直于地面。但是这种投影方式也会导致越靠近图像边缘的前景成像形变越严重。另外，采用直线投影方式的广角相机的视场角越大，那么位于图像边缘的前景成像形变就越严重。例如，图1是本申请实施例提供的一种由广角相机采用直线投影方式采集得到的图像示意图。如图1所示，图像A包括位于图像边缘区域的人体成像A1和位于图像中心区域的人体成像A2。假设人体成像A1的采集对象和人体成像A2的采集对象的人体姿态和人体外形一致，但站立位置不同。参见图1，人体成像A1相较于人体成像A2在水平方向上明显被拉伸(横向拉伸)。
另外，在广角相机相对于水平面倾斜设置，即广角相机的俯仰角不为0的情况下，采用直线投影时还会产生梯形形变。这种情况下广角相机拍摄到的图像中的前景成像除了会出现成像形变以外，还会出现成像倾斜。其中，广角相机的俯仰角可以是广角相机的主光轴与水平面的夹角。例如，图2是本申请实施例提供的另一种由广角相机采用直线投影方式采集得到的图像示意图。如图2所示，图像B包括位于图像右侧边缘区域的人体成像B1和位于图像 中心区域的人体成像B2。假设人体成像B1的采集对象和人体成像B2的采集对象的人体姿态和人体外形一致，但站立位置不同。参见图2，人体成像B1相较于人体成像B2被横向拉伸，且人体成像B1的上半身明显向右倾斜。本申请实施例中提及的人体倾斜，是指人体高度方向(例如人脸中心点与人体中心点的连线)与竖直方向存在一定夹角。
又例如，对于采用球面投影方式的广角相机，这种投影方式能够使广角相机拍摄到的图像中各个区域的前景成像不发生形变，但也会导致越靠近图像边缘的前景成像的倾斜越严重。另外，采用球面投影方式的广角相机拍摄的图像的背景畸变比较严重。例如，图3是本申请实施例提供的一种由广角相机采用球面投影方式采集得到的图像示意图。如图3所示，图像C包括位于图像右侧边缘区域的人体成像C1和位于图像中心区域的人体成像C2。假设人体成像C1的采集对象和人体成像C2的采集对象的人体姿态和人体外形一致，但站立位置不同。参见图3，人体成像C1的上半身明显向左倾斜。
又例如，帕尼尼投影方式是一种用户自定义的投影方式。帕尼尼投影方式的图像效果介于直线投影方式的图像效果和球面投影方式的图像效果之间。对于采用帕尼尼投影方式的广角相机，根据图像中的不同成像位置做不同程度的畸变矫正。主要原理是将以图像的中心点为圆心，半径为r的圆上的像素，按照实际显示效果，统一沿着半径r，向圆内或外移动距离d。编码中的表示就是一个映射表。该映射表通常按照显示效果由用户进行自定义。总体来说，图像中心处的映射变化较小(即畸变矫正程度较小)，图像边缘处的映射变化较大(即畸变矫正程度较大)。采用帕尼尼投影方式的广角相机拍摄的图像的边缘区域的前景成像会有一定的形变，但形变程度低于直线投影方式下的形变程度。并且，采用帕尼尼投影方式的广角相机拍摄的图像的边缘区域的前景成像会有一定的倾斜，但倾斜程度低于球面投影方式下的倾斜程度。例如，图4是本申请实施例提供的一种由广角相机采用帕尼尼投影方式采集得到的图像示意图。如图4所示，图像D包括位于图像右侧边缘区域的人体成像D1和位于图像中心区域的人体成像D2。假设人体成像D1的采集对象和人体成像D2的采集对象的人体姿态和人体外形一致，但站立位置不同。参见图4，人体成像D1相较于人体成像D2被横向拉伸，但拉伸程度低于图像A中人体成像A1的拉伸程度。人体成像D1的上半身略微向左倾斜，但倾斜程度低于图像C中人体成像C1的倾斜程度。
以图2示出的图像B为例，将图像B中的人体成像B1和人体成像B2裁剪后拼接到一起，可以得到如图5所示的拼接画面，也即是，该拼接画面包括图像B中的人体成像B1和人体成像B2。由于人体成像B1相较于人体成像B2被横向拉伸，且人体成像B1的上半身明显向右倾斜，因此拼接画面明显不协调，显示效果较差。
基于此，本申请实施例提供了一种成像畸变矫正方法，该方法应用于图像处理设备。图像处理设备在获取相机采集的图像之后，根据人体成像在图像中的成像位置对人体成像进行畸变矫正。由于图像中的人体成像的畸变程度受成像位置的影响，成像位置越靠近图像边缘，则人体成像的畸变程度越严重。通过针对位于图像不同位置的人体成像的不同畸变程度，分别对人体成像进行矫正，这样后续将位于图像的不同位置的、分别经过畸变矫正的人体成像拼接在一起之后，可以使得拼接画面中不同人体成像的人体形变和人体倾斜程度保持一致，提高了拼接画面的协调性，从而提高了显示效果。
下面从应用场景、方法流程、软件装置、硬件装置等多个角度，对本申请提供的技术方案进行详细介绍。
下面对本申请实施例的应用场景举例说明。
本申请实施例提供的成像畸变矫正方法可以应用于图像处理设备。该图像处理设备可以是相机，或者也可以是显示设备，又或者可以是与显示设备连接的视频服务器。显示设备内置有相机，或者，显示设备与外置的相机相连。该相机为广角相机。该相机用于对拍摄区域进行拍摄以采集得到图像。图像处理设备用于对相机采集的图像中的人体成像进行畸变矫正。进一步地，图像处理设备还可以对经过畸变矫正的多个人体成像进行拼接，以供显示设备显示包含多个人体成像的拼接图像。其中，视频服务器可以是一台服务器，或者由多台服务器组成的服务器集群，或者云计算平台等。
本申请实施例提供的成像畸变矫正方法可以应用于多种场景。在视频会议场景中，显示设备可以是会议终端，例如可以是大屏、电子白板、手机、平板电脑或智能可穿戴设备等具有显示功能的电子设备。在家庭场景或教室场景中，显示设备可以是智能电视、投影设备或虚拟现实(virtual reality，VR)设备等。
例如，图6是本申请实施例提供的一种成像畸变矫正方法涉及的应用场景示意图。该应用场景是视频会议场景。如图6所示，该应用场景包括多个会议终端601A-601C(统称为会议终端601)。多个会议终端601之间通信连接。其中，会议终端601A和会议终端601B中分别内置有广角相机(图中未示出)，会议终端601C中内置有非广角相机(图中未示出)。
可选地，请继续参见图6，该应用场景还包括视频服务器602，多个会议终端601分别与视频服务器602连接。多个会议终端601之间通过视频服务器602实现通信连接，视频服务器602例如可以是多点控制单元(multi control unit，MCU)。当然，本申请实施例也不排除不同会议终端之间直接相连的情况。
广角相机采集的图像包括畸变区域和非畸变区域。位于畸变区域的人体成像，越靠近图像边缘，则畸变程度越严重。非广角相机采集的图像中的人体成像一般不会发生畸变。例如，图7是本申请实施例提供的一种广角相机采集的图像的区域分布示意图。如图7所示，该图像包括畸变区域和非畸变区域。其中，非畸变区域可以是图像的中心区域。畸变区域可以是图像中除中心区域以外的其它区域。
可选地，会议终端601A或视频服务器602可以对会议终端601A中内置的广角相机采集的图像中的人体成像进行畸变矫正。会议终端601A或视频服务器602还可以接收会议终端601B发送的由会议终端601B内置的广角相机采集的图像，并对该图像中的人体成像进行畸变矫正。
进一步地，会议终端601A或视频服务器602还可以对来自一张或多张图像中的多个人体成像进行拼接。该多个人体成像包括发生畸变但经过畸变矫正的人体成像和/或未发生畸变的人体成像。例如，会议终端601A或视频服务器602可以对来自一个会议终端的一张图像中的多个人体成像进行拼接。或者，会议终端601A或视频服务器602也可以对来自多个会议终端的多张图像中的多个人体成像进行拼接。
可选地，会议终端601B的功能以及会议终端601C的功能可参考会议终端601A的功能，本申请实施例不再一一赘述。
本申请实施例提供的成像畸变矫正方法可以用于对广角相机采集的图像中的人体成像进行畸变矫正。类似地，本申请实施例提供的成像畸变矫正方法还可以用于对广角相机采集的图像中的其它前景成像进行畸变矫正，例如猫、狗等的成像，本申请实施例对应用场景不做限定。本申请实施例中均以对人体成像进行畸变矫正为例进行说明。
下面对本申请实施例的方法流程举例说明。
例如，图8是本申请实施例提供的一种成像畸变矫正方法的流程示意图。该方法可以应用于图像处理设备。图像处理设备例如可以是如图6所示的应用场景中的任一会议终端601或与会议终端601连接的视频服务器602。如图8所示，该方法包括：
步骤801、图像处理设备获取第一相机采集的第一图像。
第一相机为广角相机。第一相机采集的图像包括畸变区域和非畸变区域。第一图像包括位于第一图像的畸变区域的第一人体成像。位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和/或人体形变。
第一种可能情况，第一图像由第一相机采用直线投影方式采集得到，则位于第一图像的畸变区域的人体成像的畸变类型包括人体形变。例如第一图像可以是图1示出的图像A，位于第一图像的畸变区域的人体成像可以参考人体成像A1。
可选地，若第一相机相对于水平面倾斜设置，即第一相机的俯仰角不为0，则位于第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜。例如第一图像可以是图2示出的图像B，位于第一图像的畸变区域的人体成像可以参考人体成像B1。
第二种可能情况，第一图像由第一相机采用帕尼尼投影方式采集得到，则位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和人体形变。例如第一图像可以是图3示出的图像C，位于第一图像的畸变区域的人体成像可以参考人体成像C1。
第三种可能情况，第一图像由第一相机采用球面投影方式采集得到，则位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜。例如第一图像可以是图4示出的图像D，位于第一图像的畸变区域的人体成像可以参考人体成像D1。
发明人发现，以上三种可能情况均满足：人体成像在第一相机采集的图像中越靠近图像边缘，则畸变程度越严重。
可选地，第一相机可以是图像处理设备内置或与图像处理设备相连的相机，图像处理设备中预先存储有第一相机所采用的投影方式。图像处理设备获取第一图像，可以是图像处理设备通过第一相机采集第一图像。这种情况下，第一图像为本端采集的图像。例如，图像处理设备为如图6所示的应用场景中的会议终端601A，第一相机为会议终端601A内置的广角相机，第一图像由会议终端601A内置的广角相机采集得到后传输至会议终端601A的处理器。
或者，图像处理设备获取第一图像，也可以是图像处理设备接收其它设备发送的第一图像，第一相机可以是其它设备内置或与其它设备相连的相机，其它设备还需向图像处理设备发送第一相机所采用的投影方式。这种情况下，第一图像为远端采集的图像。例如，图像处理设备为如图6所示的应用场景中的会议终端601A，第一相机为会议终端601B内置的广角相机，第一图像由会议终端601B内置的广角相机采集得到后，由会议终端601B发送给会议终端601A。
步骤802、图像处理设备根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正，得到第二人体成像。
其中，第二人体成像满足：人体倾斜角度小于预设角度，且人体形变程度小于预设阈值。人体倾斜角度可以是指人体高度方向与竖直方向的夹角。
可选地，图像处理设备在获取第一图像之后，可以确定第一图像中所有人体成像的成像位置，然后根据人体成像的成像位置确定哪个或哪些人体成像位于第一图像的畸变区域。第一人体成像为位于第一图像的畸变区域的任一人体成像。
可选地，图像处理设备确定第一图像中的人体成像的成像位置的一种实现方式，包括：图像处理设备采用人脸检测算法确定第一图像中的人脸成像的成像位置，然后根据该人脸成 像的成像位置确定该人脸成像所属的人体成像在第一图像中的成像位置。具体实现时，图像处理设备可以在获取第一图像中的人脸成像的成像位置之后，对第一图像中的人脸成像区域进行拓展，得到包含人体成像的成像区域。其中，人体成像可以是人体的上半身成像，或者也可以是人体的全身成像，或者也可以是人体的头肩成像，可根据人体成像的实际用途设计所需提取的人体成像，本申请实施例对人体成像所包含的内容不做限定。例如，图9是本申请实施例提供的一种图像示意图。如图9所示，该图像包括用户P的人体成像和用户Q的人体成像。图像处理设备可以对用户P在图像中的人脸成像区域P1进行拓展，得到用户P在该图像中的人体成像区域P2。图像处理设备可以对用户Q在图像中的人脸成像区域Q1进行拓展，得到用户Q在该图像中的人体成像区域Q2。其中，人体成像区域P2和人体成像区域Q2均包含人体的上半身成像。
本申请实施例中，图像处理设备还可以采用其它实现方式确定第一图像中人体成像的成像位置。例如，图像处理设备可以对第一图像进行人体实例分割，得到第一图像中的人体掩膜，再确定人体掩膜对应在第一图像中的成像区域，该成像区域即人体成像的成像位置。本申请实施例对在图像中确定人体成像的成像位置的实现方式不做限定。
在第一相机采用的投影方式不同的情况下，第一相机采集的图像中的人体成像的畸变类型和/或畸变程度不同，本申请以下实施例针对第一相机采用三种不同投影方式的情况分别进行说明。
第一种可能情况，第一图像由第一相机采用直线投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体形变。
在第一种可能情况下，步骤802的第一种实现方式包括以下步骤8021A至步骤8023A。
在步骤8021A中，图像处理设备根据第一相机的视场角和第一相机的焦距，对第一图像进行形变矫正投影变换，得到第二图像。
可选地，形变矫正投影变换为帕尼尼投影变换或球面投影变换。通过对采用直线投影方式得到的图像进行形变矫正投影变换，可以改善图像中人体成像的人体形变，但人体成像会发生倾斜。例如，图像处理设备对如图1所示的图像A进行球面投影变换，可以得到如图4所示的图像D，其中，人体成像D2对应人体成像A2，人体成像D1对应人体成像A1。人体成像D1相较于人体成像A1，人体形变得到明显改善，但人体发生了倾斜。
本申请实施例中，图像处理设备需要获取第一相机的视场角和第一相机的焦距。对于第一图像是本端采集的图像的情况，可以在图像处理设备中预先存储第一相机的视场角和第一相机的焦距。对于第一图像是远端采集的图像的情况，第一相机的视场角和第一相机的焦距可以由远端发送给图像处理设备。例如，图像处理设备为第一会议终端，第一图像由第二会议终端内置或外接的第一相机采集后发送给第一会议终端，在视频会议开始的时候，第二会议终端可以通过网络将第一相机的视场角和第一相机的焦距发送给第一会议终端。第一会议终端存储第二会议终端对应的相机视场角和相机焦距，这样后续接收到来自第二会议终端的图像之后，可以采用对应的相机视场角和相机焦距对该图像进行处理。或者，对于第一图像是远端采集的图像的情况，图像处理设备接收到第一图像之后，可以基于第一图像估算出用于采集第一图像的第一相机的视场角，再根据估算得到的视场角，估算出第一相机的焦距。例如，图像处理设备可以采用线性回归算法，基于AlexNet(一种深度卷积神经网络)对图像对应的相机视场角进行预测，本申请实施例在此对图像对应的相机视场角的具体预测过程不再详细介绍。
在步骤8022A中，图像处理设备根据第一成像位置确定第一人体成像在第二图像中的第 二成像位置。
可选地，步骤8022A的一种实现方式，包括：图像处理设备根据第一成像位置、第一相机的视场角和第一相机的焦距，确定第一人体成像在第二图像中的第二成像位置。第一成像位置可以采用直线投影下的图像坐标表示，第二成像位置可以采用球面投影下的图像坐标表示。图像处理设备可以根据第一相机的视场角和第一相机的焦距，将直线投影下的图像坐标转换成球面投影下的图像坐标。例如，图10是本申请实施例提供的一种图像坐标变换示意图。其中，z轴表示第一相机的主光轴所在方向，x轴表示图像坐标系的宽度方向，o表示第一相机的光心。如图10所示，假设物体M到光心o的连线与第一相机的主光轴的夹角为θ，θ最大取值为第一相机的视场角的一半，第一相机的焦距为f，物体M直线投影在像平面上得到M’，M’的横坐标为ru。ru＝f*tanθ。MM’的连线与以o为圆心、f为半径的球面的交点N在像平面上的投影即M’经过球面投影转换后的横坐标rd，图像处理设备可以采用球面投影公式：rd＝2f*tan(θ/2)计算得到rd。
或者，图像处理设备在获取第一图像之后，也可以先对第一图像进行形变矫正投影变换，得到第二图像，再确定第二图像中所有人体成像的成像位置。在第二图像中确定人体成像的成像位置的方式可参考上述在第一图像中确定人体成像的成像位置的方式，本申请实施例在此不再赘述。
在步骤8023A中，图像处理设备根据第二成像位置，对从第二图像中裁剪得到的第一人体成像进行旋转矫正，得到第二人体成像。
可选地，第二人体成像垂直于地面，即第二人体成像为竖直人体成像。第二成像位置包括人脸中心点位置和人体中心点位置。步骤8022A的一种实现方式，包括：图像处理设备根据第一人体成像在第二图像中的人脸中心点位置和人体中心点位置，确定第一人体成像在第二图像中的人体倾斜角度。图像处理设备根据第一人体成像在第二图像中的人体倾斜角度，对从第二图像中裁剪得到的第一人体成像进行旋转矫正。其中，人体倾斜角度可以定义为人脸中心点到人体中心点连线相对于竖直线的角度。
可选地，图像处理设备可以对第一图像中的第一人体成像进行人脸检测和人体检测，得到第一人体成像在第一图像中的人脸中心点坐标和人体中心点坐标，再计算第一人体成像经过形变矫正投影变换之后在第二图像中的人脸中心点坐标和人体中心点坐标。或者，图像处理设备可以对第二图像中的第一人体成像进行人脸检测和人体检测，直接得到第一人体成像在第二图像中的人脸中心点坐标和人体中心点坐标。例如，第一人体成像在第二图像中的人脸中心点坐标为(m1，n1)，第一人体成像在第二图像中的人体中心点坐标为(m2，n2)，则第一人体成像在第二图像中的人体倾斜角度等于arctan((m1-m2)/(n1-n2))。在对应的相机视场角为120°的图像中，人脸位于最边缘对角线上的人体成像的人体倾斜角度约为10度。
本申请实施例中，图像处理设备在获取第一人体成像在第二图像中的成像位置之后，可以先从第二图像中裁剪得到包含第一人体成像的较大区域，人体倾斜角度越大，则需要裁剪的区域越大。在对裁剪得到的包含第一人体成像的较大区域进行旋转矫正之后，再裁剪得到包含第一人体成像的较小区域。两次裁剪的包含第一人体成像的区域可以是矩形区域，或者也可以是其它指定形状的区域，本申请实施例对裁剪区域的形状不做限定。
例如，图11是本申请实施例提供的一种对第二图像中的人体成像进行旋转矫正的示意图。如图11所示，图像处理设备首先从第二图像中裁剪得到包含倾斜人体成像的矩形区域Z1，再对包含倾斜人体成像的矩形区域Z1进行旋转矫正，得到包含竖直人体成像的矩形区域Z1，最后从包含竖直人体成像的矩形区域Z1中裁剪得到包含竖直人体成像的矩形区域Z2。 其中，矩形区域Z1的范围可以根据最终需要裁剪得到的矩形区域Z2的范围确定。例如，包含倾斜人体成像的矩形区域Z1需要顺时针旋转10度后得到包含竖直人体成像的矩形区域Z1，则图像处理设备可以计算矩形区域Z2的外接矩形的四个顶点相对于矩形区域Z2的中心点的坐标，并对这四个顶点坐标做仿射变换后(逆时针旋转10度)得到矩形区域Z1的最小范围，此时矩形区域Z1也即是矩形区域Z2的外接矩形。
本实现方式中，图像处理设备先对采用直线投影方式得到的图像进行形变矫正投影变换，以改善图像中人体成像的人体形变，但经过形变矫正投影变换的图像中的人体成像会发生倾斜。然后图像处理设备再从经过形变矫正投影变换的图像中裁剪得到发生倾斜的人体成像，并对该发生倾斜的人体成像进行旋转矫正，使得人体成像的倾斜畸变也得到改善，最终得到基本不存在畸变的人体成像。
在第一种可能情况下，步骤802的第二种实现方式包括以下步骤8021B至步骤8022B。
在步骤8021B中，图像处理设备基于预先存储的成像位置与人体形变程度的第一对应关系，根据第一成像位置确定第一人体成像在第一图像中的人体形变程度。
其中，第一对应关系与第一相机的视场角相匹配。广角相机的视场角不同，则广角相机采用直线投影方式采集得到的图像中位于畸变区域的人体成像的人体形变程度不同。一般情况下，广角相机的视场角越大，则位于广角相机采集的图像的畸变区域内的前景成像的形变程度越严重。与第一相机的视场角相匹配的第一对应关系，是指在第一相机的视场角下所采集的图像中，成像位置与人体形变程度的对应关系。
在第一相机相对于地面垂直设置，即第一相机的主光轴平行于地面的情况下，第一相机采用直线投影方式采集得到的图像中的人体成像在水平方向上会被拉伸。拉伸程度由人体成像在图像中的位置决定。人体成像越靠近图像边缘，拉伸程度越严重。假设图像的宽度是w，人脸在图像中的位置到图像中心的距离是p，图像最边缘人脸的拉伸程度是α，那么p处人脸的拉伸程度β，采用线性插值的方式计算就是：β＝α*[2*p/w]。其中，图像最边缘人脸的拉伸程度α由相机视场角决定，相机视场角越大，则α取值越大，α>1。例如在视场角为80度的相机采集的图像中，图像最边缘人脸会被水平拉伸放大20％，此时α可取值为1.2。与相机视场角相匹配的成像位置与人体形变程度的对应关系可以采用该线性插值计算公式来表示。
在步骤8022B中，图像处理设备根据第一人体成像在第一图像中的人体形变程度，对从第一图像中裁剪得到的第一人体成像进行形变矫正。
本实现方式中，图像处理设备中可以预先存储有一组或多组成像位置与人体形变程度的对应关系，每组对应关系与一个相机视场角相匹配。这样，图像处理设备可以基于与图像对应的相机视场角相匹配的成像位置与人体形变程度的对应关系，根据人体成像在该图像中的成像位置确定人体成像在该图像中的人体形变程度，在从该图像中裁剪得到人体成像之后，再对裁剪得到的人体成像进行形变矫正，可以改善人体成像的人体形变。另外，本实现方式无需进行形变矫正投影变换，计算量小，对图像处理设备的处理性能要求较低。
可选地，若第一相机相对于水平面倾斜设置，则第一图像会发生梯形形变，位于第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜。也即是在这种情况下，采用直线投影方式采集得到的第一图像中的第一人体成像的畸变类型包括人体形变和人体倾斜。
对于上述第一种可能情况下步骤802的第一种实现方式，人体成像的人体形变可通过形变矫正投影变换(步骤8021A)改善，人体成像的人体倾斜可通过旋转矫正(步骤8023A)改善，因此该实现方式下，图像处理设备可以无需针对由于相机相对于水平面倾斜设置导致的 梯形形变问题。
而对于上述第一种可能情况下步骤802的第二种实现方式，除了要对第一图像中的第一人体成像进行形变矫正以外，还需要对第一图像中的第一人体成像进行旋转矫正。该实现方式下，本申请实施例提供了以下三种方案对人体成像进行旋转矫正。
本申请实施例中，图像处理设备可以先对人体成像进行旋转矫正，再对经过旋转矫正的人体成像进行形变矫正。当然也不排除图像处理设备先对人体成像进行形变矫正，再对经过形变矫正的人体成像进行旋转矫正的可能性。
第一种方案，第一成像位置包括人脸中心点位置和人体中心点位置。图像处理设备根据第一人体成像在第一图像中的人脸中心点位置和人体中心点位置，确定第一人体成像在第一图像中的人体倾斜角度。图像处理设备根据第一人体成像在第一图像中的人体倾斜角度，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。其中，人体倾斜角度可以定义为人脸中心点到人体中心点连线相对于竖直线的角度。该方案的具体实现方式可参考上述步骤8023A的实现方式，本申请实施例在此不再赘述。
第二种方案，图像处理设备基于预先存储的成像位置与人体倾斜角度的第二对应关系，确定第一人体成像在第一图像中的人体倾斜角度。图像处理设备根据第一人体成像在第一图像中的人体倾斜角度，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。
其中，第二对应关系与第一相机的视场角和第一相机的俯仰角相匹配。广角相机的视场角不同，和/或，广角相机的俯仰角不同，则广角相机采用直线投影方式采集得到的图像中位于畸变区域的人体成像的人体倾斜角度不同。一般情况下，广角相机的视场角和俯仰角越大，则位于广角相机采集的图像的畸变区域内的前景成像的倾斜角度越大。与第一相机的视场角和第一相机的俯仰角相匹配的第二对应关系，是指在第一相机的视场角和俯仰角下所采集的图像中，成像位置与人体倾斜角度的对应关系。例如对于视场角为80度的相机，如果相机的俯仰角为5度，即相机相较于水平面向下倾斜5度设置，那么在该相机采集的图像中，图像最边缘的人体成像的人体倾斜角度约为10度。如果相机的俯仰角为15度，即相机相较于水平面向下倾斜15度设置，那么在该相机采集的图像中，图像最边缘的人体成像的人体倾斜角度约为20度。
可选地，相机的俯仰角可以是在相机部署完成之后由人工输入图像处理设备的，或者也可以由设备采用硬件或软件的方式自行测量或计算。例如，硬件上可以在相机或相机所部署在的设备上集成能够判断设备放置角度的传感器，比如加速度传感器或重力传感器等。软件上可以由图像处理设备基于相机拍摄得到的图像，计算拍摄场景中的竖直线的倾斜程度，拍摄场景中的竖直线比如可以是窗户、电视或投影幕布的左右边缘线。具体实现时，图像处理设备可以先对图像进行目标检测分类和分割，再基于分割结果判断相机的倾斜方向，并基于分割结果提取目标的边缘线以计算边缘线相对于竖直线的倾斜角度，进而得到相机的俯仰角。本申请实施例对图像处理设备获取相机俯仰角的方式不做限定。
本方案中，图像处理设备中可以预先存储有一组或多组成像位置与人体倾斜角度的对应关系，每组对应关系与一个相机视场角和一个相机俯仰角相匹配。这样，图像处理设备可以基于与图像对应的相机视场角和相机俯仰角相匹配的成像位置与人体倾斜角度的对应关系，根据人体成像在该图像中的成像位置确定人体成像在该图像中的人体倾斜角度，在从该图像中裁剪得到人体成像之后，先对裁剪得到的该人体成像进行旋转矫正，再进一步对经过旋转矫正得到人体成像进行形变矫正，可以改善人体成像的人体倾斜和人体形变。另外，本方案无需计算人体成像的人体倾斜角度，计算量小，对图像处理设备的处理性能要求较低。
在上述第一种方案和第二种方案中，图像处理设备需要先从相机采集的原始图像(例如第一图像)中裁剪得到人体成像，再对裁剪得到的人体成像进行旋转矫正和形变矫正。其中，从原始图像中裁剪人体成像的实现方式可参考上述步骤8023A中的相关描述，本申请实施例在此不再赘述。
第三种方案，图像处理设备采用与第一相机的俯仰角对应的梯形变换矩阵对第一图像进行梯形变换，得到第三图像。然后图像处理设备根据第一人体成像在第三图像中的第三成像位置，对第一人体成像进行畸变矫正。
其中，梯形变换是一种透视变换，用于建立两个平面场之间的对应关系。对图像进行梯形变换，也即是将图像从一个视平面投影到另一个视平面。在已知相机的俯仰角之后，就知道了图像需要矫正的程度。通过向图像处理设备输入原始图像中多个点(大于或等于4个点)对应的相机坐标以及希望矫正后得到的目标图像中该多个点对应的相机坐标，使得图像处理设备可以求解得到梯形变换矩阵。其中，相机坐标是指相机坐标系下的三维坐标。相机坐标系是以相机的光心为原点，以主光轴为z轴建立的三维直角坐标系。其中，相机坐标系的x轴与该相机采集的图像对应的图像坐标系的x轴平行。相机坐标系的y轴与该相机采集的图像对应的图像坐标系的y轴平行。图像处理设备可以根据多个点在原始图像中对应的相机坐标和在目标图像中对应的相机坐标，基于以下公式求解得到3*3的梯形变换矩阵H。
目标图像＝原始图像*H。
本方案中，由于在相机相对于水平面倾斜设置的情况下，相机采集的图像会发生梯形形变，本方案通过对相机采集到的图像先进行梯形变换，以改善图像的梯形形变，进一步再对图像中的人体成像进行形变矫正，可以提高最终获取的人体成像的显示效果。
上述第三种方案也可以结合上述第一种可能情况下步骤802的第一种实现方式使用，也即是，在步骤8021A执行之前，图像处理设备先对第一图像进行梯形变换得到第三图像，后续图像处理设备针对第三图像进行进一步的图像处理。上述图像处理设备根据第一人体成像在第三图像中的第三成像位置，对第一人体成像进行畸变矫正的实现方式，可以包括：图像处理设备根据第三成像位置，对从第三图像中裁剪得到的第一人体成像进行水平缩放。
第二种可能情况，第一图像由第一相机采用帕尼尼投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和人体形变。步骤802的一种实现方式包括：图像处理设备根据第一相机的视场角和第一相机的焦距，对第一图像进行球面投影变换，得到第四图像。图像处理设备根据第一成像位置确定第一人体成像在第四图像中的第四成像位置。图像处理设备根据第四成像位置，对从第四图像中裁剪得到的第一人体成像进行旋转矫正，得到第二人体成像。
可选地，图像处理设备可以根据第一相机的视场角和第一相机的焦距，先对采用帕尼尼投影方式采集得到第一图像进行直线投影变换，再对经过直线投影变换的图像进行球面投影变换，得到第四图像。图像处理设备根据第四成像位置，对从第四图像中裁剪得到的第一人体成像进行旋转矫正的实现方式，可参考上述步骤8023A中，图像处理设备根据第二成像位置，对从第二图像中裁剪得到的第一人体成像进行旋转矫正的实现方式，本申请实施例在此不再赘述。
第三种可能情况，第一图像由第一相机采用球面投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜。步骤802的一种实现方式包括：图像处理设备根据第一成像位置，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。图像处理设备根据第一成像位置，对从第一图像中裁剪得到的第一人体成像进行旋转矫正的实现方式，可 参考上述第一种方案和第二种方案中的相关实现方式，本申请实施例在此不再赘述。
本申请实施例中，图像处理设备可以针对广角相机采用的不同投影方式，分别采用对应的策略改善图像中人体成像的人体形变和/或人体倾斜问题。经过畸变矫正的人体成像可以用于多种需要拼接显示的显示模式场景下。可选地，图像处理设备在得到经过畸变矫正的人体成像之后，还可以执行以下步骤803至步骤805。
步骤803、图像处理设备获取第三人体成像。
第三人体成像为从目标图像的非畸变区域中裁剪得到的人体成像，或者，第三人体成像为位于目标图像的畸变区域、且经过畸变矫正的人体成像。
第一种实施场景中，目标图像为第一图像，也即是，第一人体成像和第三人体成像来自同一图像。例如，第一人体成像为如图1所示的图像A中的人体成像A1，第三人体成像为如图1所示的图像A中的人体成像A2(未发生畸变的人体成像)。
第二种实施场景中，目标图像为除第一图像以外的其它图像，也即是，第一人体成像和第三人体成像来自不同图像。例如，第一人体成像为如图1所示的图像A中的人体成像A1，第三人体成像为如图3所示的图像C中的人体成像C1经过畸变矫正后的人体成像。
可选地，目标图像为第二相机采集的图像，第二相机与第一相机不同，也即是，第一人体成像和第三人体成像分别来自不同相机采集的图像。可选地，第二相机对目标图像的采集时刻与第一相机对第一图像的采集时刻相同。
步骤804、图像处理设备对第二人体成像和第三人体成像进行拼接，得到拼接图像。
基于上述步骤801和步骤802可知，第二人体成像基于来自第一图像的第一人体成像经过畸变矫正得到。
结合步骤803的第一种实施场景，第三人体成像为从第一图像的非畸变区域中裁剪得到的人体成像，或者，第三人体成像为位于第一图像的畸变区域、且经过畸变矫正的人体成像。图像处理设备对第二人体成像和第三人体成像进行拼接，也即是对来自同一图像的不同人体成像进行拼接，由于用于拼接的人体成像经过畸变矫正或在原始图像中就未发生畸变，因此在将多个人体成像拼接在一起之后，可以使得拼接画面中不同人体成像的人体形变和人体倾斜程度基本保持一致，拼接画面的协调性较好，进而可以保证显示效果。
结合步骤803的第二种实施场景，第三人体成像为从除第一图像以外的其它图像的非畸变区域中裁剪得到的人体成像，或者，第三人体成像为位于除第一图像以外的其它图像的畸变区域、且经过畸变矫正的人体成像。图像处理设备对第二人体成像和第三人体成像进行拼接，也即是对来自不同图像的人体成像进行拼接。在视频会议场景中，多个会议终端在同一时刻分别采集视频会议图像，图像处理设备可以对来自不同视频会议图像的人体成像进行拼接，通过将不同会议室的人放到同一个拼接画面中，可以让多个参会方感觉像是在一起工作，从而能够改善多人协作氛围。
可选地，响应于接收到对目标显示模式的选择指令，图像处理设备开始执行上述步骤802。目标显示模式可以是智能均分模式或画廊模式。可选地，图像处理设备按照目标显示模式对第二人体成像和/或第三人体成像执行缩放操作，并对经过缩放操作的第二人体成像和第三人体成像进行拼接，得到拼接图像。
智能均分模式下，图像处理设备可以截取包含人体成像的矩形区域，并通过缩放操作使每个矩形区域的高度等于拼接画面的高度。其中，智能均分模式下，图像处理设备可以统一截取人体的全身成像或人体的上半身成像。例如，图12是本申请实施例提供的一种智能均分模式下的拼接图像示意图。如图12所示，该拼接图像由4个包含人体的全身成像的矩形区域 拼接得到，且不同人体成像所在矩形区域的高度均相同。
画廊模式下，图像处理设备可以截取包含人体成像的矩形区域，相较于智能均分模式，图像处理设备可以在画廊模式下截取范围稍小的人体成像，比如图像处理设备可以截取人体的上半身成像和/或头肩成像。画廊模式下的图像拼接方式可以有多种，用户可根据显示需求自行选择。例如，图13至图15分别是本申请实施例提供的一种画廊模式下的拼接图像示意图。如图13所示，该拼接图像由4个包含人体的头肩成像的矩形区域采用四宫格形式拼接得到。如图14所示，该拼接图像由9个包含人体的头肩成像的矩形区域采用九宫格形式拼接得到。如图15所示，该拼接图像由5个包含人体的头肩成像的矩形区域拼接得到，其中4个人体成像为小图，1个人体成像为大图。
步骤805、图像处理设备输出拼接图像到屏幕上显示。
可选地，图像处理设备输出拼接图像到屏幕上显示，可以是自身屏幕上显示拼接图像，或者也可以是向其它设备发送拼接图像以供其它设备显示。例如视频会议场景中，会议终端可以基于本端采集的图像对多个与会人员的人体成像进行拼接显示，这样可以将座位比较分散的与会人员集中显示在拼接画面中。会议终端还可以向远端的其它会议终端发送拼接图像，以供其它会议终端显示。
综上所述，在本申请实施例提供的成像畸变矫正方法中，图像处理设备在获取相机采集的图像之后，根据人体成像在图像中的成像位置对人体成像进行畸变矫正。由于图像中的人体成像的畸变程度受成像位置的影响，成像位置越靠近图像边缘，则人体成像的畸变程度越严重。通过针对位于图像不同位置的人体成像的不同畸变程度，分别对人体成像进行矫正，这样后续将位于图像的不同位置的、分别经过畸变矫正的人体成像拼接在一起之后，可以使得拼接画面中不同人体成像的人体形变和人体倾斜程度保持一致，提高了拼接画面的协调性，从而提高了显示效果。
本申请实施例提供的成像畸变矫正方法的步骤的先后顺序能够进行适当调整，步骤也能够根据情况进行相应增减。任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化的方法，都应涵盖在本申请的保护范围之内。例如，图像处理设备在执行上述步骤802之后，可以将得到的经过畸变矫正的人体成像发送给其它设备，以供其它设备进行拼接显示。本申请实施例不再一一赘述。
下面对本申请实施例的虚拟装置举例说明。
例如，图16是本申请实施例提供的一种成像畸变矫正装置的结构示意图。如图16所示，成像畸变矫正装置1600包括：第一获取模块1601和畸变矫正模块1602。
第一获取模块1601，用于获取第一相机采集的第一图像，第一图像包括位于第一图像的畸变区域的第一人体成像，位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和/或人体形变。
畸变矫正模块1602，用于根据第一人体成像在第一图像中的第一成像位置，对第一人体成像进行畸变矫正，得到第二人体成像。
可选地，第一图像由第一相机采用直线投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体形变。
可选地，畸变矫正模块1602，用于：根据第一相机的视场角和第一相机的焦距，对第一图像进行形变矫正投影变换，得到第二图像。根据第一成像位置确定第一人体成像在第二图像中的第二成像位置。根据第二成像位置，对从第二图像中裁剪得到的第一人体成像进行旋 转矫正。
可选地，畸变矫正模块1602，用于：根据第一成像位置、第一相机的视场角和第一相机的焦距，确定第一人体成像在第二图像中的第二成像位置。
可选地，第二成像位置包括人脸中心点位置和人体中心点位置，畸变矫正模块1602，用于：根据人脸中心点位置和人体中心点位置，确定第一人体成像在第二图像中的人体倾斜角度。根据第一人体成像在第二图像中的人体倾斜角度，对从第二图像中裁剪得到的第一人体成像进行旋转矫正。
可选地，畸变矫正模块1602，用于：基于预先存储的成像位置与人体形变程度的第一对应关系，根据第一成像位置确定第一人体成像在第一图像中的人体形变程度，其中，第一对应关系与第一相机的视场角相匹配。根据第一人体成像在第一图像中的人体形变程度，对从第一图像中裁剪得到的第一人体成像进行形变矫正。
可选地，第一相机相对于水平面倾斜设置，位于第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜，第一成像位置包括人脸中心点位置和人体中心点位置。畸变矫正模块1602，用于：根据人脸中心点位置和人体中心点位置，确定第一人体成像在第一图像中的人体倾斜角度。根据第一人体成像在第一图像中的人体倾斜角度，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。
可选地，第一相机相对于水平面倾斜设置，位于第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜，畸变矫正模块1602，还用于：采用与第一相机的俯仰角对应的梯形变换矩阵对第一图像进行梯形变换，得到第三图像。根据第一人体成像在第三图像中的第三成像位置，对第一人体成像进行畸变矫正。
可选地，畸变矫正模块1602，用于：根据第三成像位置，对从第三图像中裁剪得到的第一人体成像进行水平缩放。
可选地，第一图像由第一相机采用帕尼尼投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和人体形变。畸变矫正模块1602，用于：根据第一相机的视场角和第一相机的焦距，对第一图像进行球面投影变换，得到第四图像。根据第一成像位置确定第一人体成像在第四图像中的第四成像位置。根据第四成像位置，对从第四图像中裁剪得到的第一人体成像进行旋转矫正。
可选地，第一图像由第一相机采用球面投影方式采集得到，位于第一图像的畸变区域的人体成像的畸变类型包括人体倾斜。畸变矫正模块1602，用于：根据第一成像位置，对从第一图像中裁剪得到的第一人体成像进行旋转矫正。
可选地，如图17所示，成像畸变矫正装置1600还包括：第二获取模块1603、拼接模块1604和输出模块1605。第二获取模块1603，用于获取第三人体成像，第三人体成像为从目标图像的非畸变区域中裁剪得到的人体成像，或者，第三人体成像为位于目标图像的畸变区域、且经过畸变矫正的人体成像。拼接模块1604，用于对第二人体成像和第三人体成像进行拼接，得到拼接图像。输出模块1605，用于输出拼接图像到屏幕上显示。
可选地，目标图像为第一图像，或者，目标图像为第二相机采集的图像，第二相机与第一相机不同。
可选地，畸变矫正模块1602，用于响应于接收到对目标显示模式的选择指令，根据第一成像位置，对第一人体成像进行畸变矫正。拼接模块1604，用于按照目标显示模式对第二人体成像和/或第三人体成像执行缩放操作，并对经过缩放操作的第二人体成像和第三人体成像进行拼接，得到拼接图像。
关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。
下面对本申请实施例涉及的基本硬件结构举例说明。
例如，图18是本申请实施例提供的一种成像畸变矫正装置的硬件结构示意图。如图18所示，成像畸变矫正装置1800包括处理器1801和存储器1802，存储器1801与存储器1802通过总线1803连接。图18以处理器1801和存储器1802相互独立说明。可选地，处理器1801和存储器1802集成在一起。可选地，结合图6来看，图18中的成像畸变矫正装置1800是图6所示的应用场景中的任一会议终端601或视频服务器602。
其中，存储器1802用于存储计算机程序，计算机程序包括操作系统和程序代码。存储器1802是各种类型的存储介质，例如只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、电可擦可编程只读存储器(electrically erasable programmable read-only memory，EEPROM)、只读光盘(compact disc read-only memory，CD-ROM)、闪存、光存储器、寄存器、光盘存储、光碟存储、磁盘或者其它磁存储设备。
其中，处理器1801是通用处理器或专用处理器。处理器1801可能是单核处理器或多核处理器。处理器1801包括至少一个电路，以执行本申请实施例提供的上述成像畸变矫正方法。
可选地，成像畸变矫正装置1800还包括网络接口1804，网络接口1804通过总线1803与处理器1801和存储器1802连接。网络接口1804能够实现成像畸变矫正装置1800与其它设备通信。例如，处理器1801能够通过网络接口1804与其它设备通信来获取相机采集的图像等。
可选地，成像畸变矫正装置1800还包括输入/输出(input/output，I/O)接口1805，I/O接口1805通过总线1803与处理器1801和存储器1802连接。处理器1801能够通过I/O接口1805接收输入的命令或数据等。I/O接口1805用于成像畸变矫正装置1800连接输入设备，这些输入设备例如是键盘、鼠标等。可选地，在一些可能的场景中，上述网络接口1804和I/O接口1805被统称为通信接口。
可选地，成像畸变矫正装置1800还包括显示器1806，显示器1806通过总线1803与处理器1801和存储器1802连接。显示器1806能够用于显示处理器1801执行上述方法产生的中间结果和/或最终结果等，例如显示拼接图像。在一种可能的实现方式中，显示器1806是触控显示屏，以提供人机交互接口。
其中，总线1803是任何类型的，用于实现成像畸变矫正装置1800的内部器件互连的通信总线。例如系统总线。本申请实施例以成像畸变矫正装置1800内部的上述器件通过总线1803互连为例说明，可选地，成像畸变矫正装置1800内部的上述器件采用除了总线1803之外的其他连接方式彼此通信连接，例如成像畸变矫正装置1800内部的上述器件通过成像畸变矫正装置1800内部的逻辑接口互连。
上述器件可以分别设置在彼此独立的芯片上，也可以至少部分的或者全部的设置在同一块芯片上。将各个器件独立设置在不同的芯片上，还是整合设置在一个或者多个芯片上，往往取决于产品设计的需要。本申请实施例对上述器件的具体实现形式不做限定。
图18所示的成像畸变矫正装置1800仅仅是示例性的，在实现过程中，成像畸变矫正装置1800包括其他组件，本文不再一一列举。图18所示的成像畸变矫正装置1800可以通过执行上述实施例提供的方法的全部或部分步骤来实现对人体成像的畸变矫正。
本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有指令，当所述指令被处理器执行时，实现如图6所示的成像畸变矫正方法。
本申请实施例还提供了一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时，实现如图6所示的成像畸变矫正方法。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
在本申请实施例中，术语“第一”、“第二”和“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性。
本申请中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
需要说明的是，本申请所涉及的信息(包括但不限于用户设备信息、用户个人信息等)、数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)以及信号，均为经用户授权或者经过各方充分授权的，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。例如，本申请中涉及到的图像数据等都是在充分授权的情况下获取的。
以上所述仅为本申请的可选实施例，并不用以限制本申请，凡在本申请的构思和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (31)

  1. 一种成像畸变矫正方法，其特征在于，所述方法包括：
    获取第一相机采集的第一图像，所述第一图像包括位于所述第一图像的畸变区域的第一人体成像，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和/或人体形变；
    根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，得到第二人体成像。
  2. 根据权利要求1所述的方法，其特征在于，所述第一图像由所述第一相机采用直线投影方式采集得到，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体形变。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，包括：
    根据所述第一相机的视场角和所述第一相机的焦距，对所述第一图像进行形变矫正投影变换，得到第二图像；
    根据所述第一成像位置确定所述第一人体成像在所述第二图像中的第二成像位置；
    根据所述第二成像位置，对从所述第二图像中裁剪得到的所述第一人体成像进行旋转矫正。
  4. 根据权利要求3所述的方法，其特征在于，所述根据所述第一成像位置确定所述第一人体成像在所述第二图像中的第二成像位置，包括：
    根据所述第一成像位置、所述第一相机的视场角和所述第一相机的焦距，确定所述第一人体成像在所述第二图像中的第二成像位置。
  5. 根据权利要求3或4所述的方法，其特征在于，所述第二成像位置包括人脸中心点位置和人体中心点位置，所述根据所述第二成像位置，对从所述第二图像中裁剪得到的所述第一人体成像进行旋转矫正，包括：
    根据所述人脸中心点位置和所述人体中心点位置，确定所述第一人体成像在所述第二图像中的人体倾斜角度；
    根据所述第一人体成像在所述第二图像中的人体倾斜角度，对从所述第二图像中裁剪得到的所述第一人体成像进行旋转矫正。
  6. 根据权利要求2所述的方法，其特征在于，所述根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，包括：
    基于预先存储的成像位置与人体形变程度的第一对应关系，根据所述第一成像位置确定所述第一人体成像在所述第一图像中的人体形变程度，其中，所述第一对应关系与所述第一相机的视场角相匹配；
    根据所述第一人体成像在所述第一图像中的人体形变程度，对从所述第一图像中裁剪得到的所述第一人体成像进行形变矫正。
  7. 根据权利要求6所述的方法，其特征在于，所述第一相机相对于水平面倾斜设置，位于所述第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜，所述第一成像位置包括人脸中心点位置和人体中心点位置，所述根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，还包括：
    根据所述人脸中心点位置和所述人体中心点位置，确定所述第一人体成像在所述第一图像中的人体倾斜角度；
    根据所述第一人体成像在所述第一图像中的人体倾斜角度，对从所述第一图像中裁剪得到的所述第一人体成像进行旋转矫正。
  8. 根据权利要求2至6任一所述的方法，其特征在于，所述第一相机相对于水平面倾斜设置，位于所述第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜，在所述获取第一相机采集的第一图像之后，所述方法还包括：
    采用与所述第一相机的俯仰角对应的梯形变换矩阵对所述第一图像进行梯形变换，得到第三图像；
    所述根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，包括：
    根据所述第一人体成像在所述第三图像中的第三成像位置，对所述第一人体成像进行畸变矫正。
  9. 根据权利要求8所述的方法，其特征在于，所述根据所述第一人体成像在所述第三图像中的第三成像位置，对所述第一人体成像进行畸变矫正，包括：
    根据所述第三成像位置，对从所述第三图像中裁剪得到的所述第一人体成像进行水平缩放。
  10. 根据权利要求1所述的方法，其特征在于，所述第一图像由所述第一相机采用帕尼尼投影方式采集得到，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和人体形变，所述根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，包括：
    根据所述第一相机的视场角和所述第一相机的焦距，对所述第一图像进行球面投影变换，得到第四图像；
    根据所述第一成像位置确定所述第一人体成像在所述第四图像中的第四成像位置；
    根据所述第四成像位置，对从所述第四图像中裁剪得到的所述第一人体成像进行旋转矫正。
  11. 根据权利要求1所述的方法，其特征在于，所述第一图像由所述第一相机采用球面投影方式采集得到，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体倾斜，所述根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，包括：
    根据所述第一成像位置，对从所述第一图像中裁剪得到的所述第一人体成像进行旋转矫正。
  12. 根据权利要求1至11任一所述的方法，其特征在于，所述方法还包括：
    获取第三人体成像，所述第三人体成像为从目标图像的非畸变区域中裁剪得到的人体成像，或者，所述第三人体成像为位于目标图像的畸变区域、且经过畸变矫正的人体成像；
    对所述第二人体成像和所述第三人体成像进行拼接，得到拼接图像；
    输出所述拼接图像到屏幕上显示。
  13. 根据权利要求12所述的方法，其特征在于，所述目标图像为所述第一图像，或者，所述目标图像为第二相机采集的图像，所述第二相机与所述第一相机不同。
  14. 根据权利要求12或13所述的方法，其特征在于，所述根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，包括：
    响应于接收到对目标显示模式的选择指令，根据所述第一成像位置，对所述第一人体成像进行畸变矫正；
    所述对所述第二人体成像和所述第三人体成像进行拼接，得到拼接图像，包括：
    按照所述目标显示模式对所述第二人体成像和/或所述第三人体成像执行缩放操作，并对经过缩放操作的第二人体成像和第三人体成像进行拼接，得到所述拼接图像。
  15. 一种成像畸变矫正装置，其特征在于，所述装置包括：
    第一获取模块，用于获取第一相机采集的第一图像，所述第一图像包括位于所述第一图像的畸变区域的第一人体成像，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和/或人体形变；
    畸变矫正模块，用于根据所述第一人体成像在所述第一图像中的第一成像位置，对所述第一人体成像进行畸变矫正，得到第二人体成像。
  16. 根据权利要求15所述的装置，其特征在于，所述第一图像由所述第一相机采用直线投影方式采集得到，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体形变。
  17. 根据权利要求16所述的装置，其特征在于，所述畸变矫正模块，用于：
    根据所述第一相机的视场角和所述第一相机的焦距，对所述第一图像进行形变矫正投影变换，得到第二图像；
    根据所述第一成像位置确定所述第一人体成像在所述第二图像中的第二成像位置；
    根据所述第二成像位置，对从所述第二图像中裁剪得到的所述第一人体成像进行旋转矫正。
  18. 根据权利要求17所述的装置，其特征在于，所述畸变矫正模块，用于：
    根据所述第一成像位置、所述第一相机的视场角和所述第一相机的焦距，确定所述第一人体成像在所述第二图像中的第二成像位置。
  19. 根据权利要求17或18所述的装置，其特征在于，所述第二成像位置包括人脸中心点 位置和人体中心点位置，所述畸变矫正模块，用于：
    根据所述人脸中心点位置和所述人体中心点位置，确定所述第一人体成像在所述第二图像中的人体倾斜角度；
    根据所述第一人体成像在所述第二图像中的人体倾斜角度，对从所述第二图像中裁剪得到的所述第一人体成像进行旋转矫正。
  20. 根据权利要求16所述的装置，其特征在于，所述畸变矫正模块，用于：
    基于预先存储的成像位置与人体形变程度的第一对应关系，根据所述第一成像位置确定所述第一人体成像在所述第一图像中的人体形变程度，其中，所述第一对应关系与所述第一相机的视场角相匹配；
    根据所述第一人体成像在所述第一图像中的人体形变程度，对从所述第一图像中裁剪得到的所述第一人体成像进行形变矫正。
  21. 根据权利要求20所述的装置，其特征在于，所述第一相机相对于水平面倾斜设置，位于所述第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜，所述第一成像位置包括人脸中心点位置和人体中心点位置，所述畸变矫正模块，用于：
    根据所述人脸中心点位置和所述人体中心点位置，确定所述第一人体成像在所述第一图像中的人体倾斜角度；
    根据所述第一人体成像在所述第一图像中的人体倾斜角度，对从所述第一图像中裁剪得到的所述第一人体成像进行旋转矫正。
  22. 根据权利要求16至20任一所述的装置，其特征在于，所述第一相机相对于水平面倾斜设置，位于所述第一图像的畸变区域的人体成像的畸变类型还包括人体倾斜，所述畸变矫正模块，还用于：
    采用与所述第一相机的俯仰角对应的梯形变换矩阵对所述第一图像进行梯形变换，得到第三图像；
    根据所述第一人体成像在所述第三图像中的第三成像位置，对所述第一人体成像进行畸变矫正。
  23. 根据权利要求22所述的装置，其特征在于，所述畸变矫正模块，用于：
    根据所述第三成像位置，对从所述第三图像中裁剪得到的所述第一人体成像进行水平缩放。
  24. 根据权利要求15所述的装置，其特征在于，所述第一图像由所述第一相机采用帕尼尼投影方式采集得到，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体倾斜和人体形变，所述畸变矫正模块，用于：
    根据所述第一相机的视场角和所述第一相机的焦距，对所述第一图像进行球面投影变换，得到第四图像；
    根据所述第一成像位置确定所述第一人体成像在所述第四图像中的第四成像位置；
    根据所述第四成像位置，对从所述第四图像中裁剪得到的所述第一人体成像进行旋转矫 正。
  25. 根据权利要求15所述的装置，其特征在于，所述第一图像由所述第一相机采用球面投影方式采集得到，位于所述第一图像的畸变区域的人体成像的畸变类型包括人体倾斜，所述畸变矫正模块，用于：
    根据所述第一成像位置，对从所述第一图像中裁剪得到的所述第一人体成像进行旋转矫正。
  26. 根据权利要求15至25任一所述的装置，其特征在于，所述装置还包括：
    第二获取模块，用于获取第三人体成像，所述第三人体成像为从目标图像的非畸变区域中裁剪得到的人体成像，或者，所述第三人体成像为位于目标图像的畸变区域、且经过畸变矫正的人体成像；
    拼接模块，用于对所述第二人体成像和所述第三人体成像进行拼接，得到拼接图像；
    输出模块，用于输出所述拼接图像到屏幕上显示。
  27. 根据权利要求26所述的装置，其特征在于，所述目标图像为所述第一图像，或者，所述目标图像为第二相机采集的图像，所述第二相机与所述第一相机不同。
  28. 根据权利要求26或27所述的装置，其特征在于，
    所述畸变矫正模块，用于响应于接收到对目标显示模式的选择指令，根据所述第一成像位置，对所述第一人体成像进行畸变矫正；
    所述拼接模块，用于按照所述目标显示模式对所述第二人体成像和/或所述第三人体成像执行缩放操作，并对经过缩放操作的第二人体成像和第三人体成像进行拼接，得到所述拼接图像。
  29. 一种成像畸变矫正装置，其特征在于，包括：处理器和存储器；
    所述存储器，用于存储计算机程序，所述计算机程序包括程序指令；
    所述处理器，用于调用所述计算机程序，实现如权利要求1至14任一所述的成像畸变矫正方法。
  30. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有指令，当所述指令被处理器执行时，实现如权利要求1至14任一所述的成像畸变矫正方法。
  31. 一种计算机程序产品，其特征在于，包括计算机程序，所述计算机程序被处理器执行时，实现如权利要求1至14任一所述的成像畸变矫正方法。
PCT/CN2023/083906 2022-06-30 2023-03-25 成像畸变矫正方法及装置 WO2024001342A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202210772678.6 2022-06-30
CN202210772678 2022-06-30
CN202211044984.4 2022-08-30
CN202211044984.4A CN117395518A (zh) 2022-06-30 2022-08-30 成像畸变矫正方法及装置

Publications (1)

Publication Number Publication Date
WO2024001342A1 true WO2024001342A1 (zh) 2024-01-04

Family

ID=89382665

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/083906 WO2024001342A1 (zh) 2022-06-30 2023-03-25 成像畸变矫正方法及装置

Country Status (1)

Country Link
WO (1) WO2024001342A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120093365A1 (en) * 2010-10-15 2012-04-19 Dai Nippon Printing Co., Ltd. Conference system, monitoring system, image processing apparatus, image processing method and a non-transitory computer-readable storage medium
CN110555813A (zh) * 2019-08-27 2019-12-10 成都数之联科技有限公司 一种无人机遥感影像的快速几何校正方法及系统
CN111105367A (zh) * 2019-12-09 2020-05-05 Oppo广东移动通信有限公司 人脸畸变校正方法、装置、电子设备及存储介质
CN113487500A (zh) * 2021-06-28 2021-10-08 北京紫光展锐通信技术有限公司 图像畸变校正方法与装置、电子设备和存储介质
CN114079745A (zh) * 2020-08-19 2022-02-22 信骅科技股份有限公司 适用于具广角网络摄影机的视频会议系统的图像显示方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120093365A1 (en) * 2010-10-15 2012-04-19 Dai Nippon Printing Co., Ltd. Conference system, monitoring system, image processing apparatus, image processing method and a non-transitory computer-readable storage medium
CN110555813A (zh) * 2019-08-27 2019-12-10 成都数之联科技有限公司 一种无人机遥感影像的快速几何校正方法及系统
CN111105367A (zh) * 2019-12-09 2020-05-05 Oppo广东移动通信有限公司 人脸畸变校正方法、装置、电子设备及存储介质
CN114079745A (zh) * 2020-08-19 2022-02-22 信骅科技股份有限公司 适用于具广角网络摄影机的视频会议系统的图像显示方法
CN113487500A (zh) * 2021-06-28 2021-10-08 北京紫光展锐通信技术有限公司 图像畸变校正方法与装置、电子设备和存储介质

Similar Documents

Publication Publication Date Title
US11748906B2 (en) Gaze point calculation method, apparatus and device
WO2021208371A1 (zh) 多摄变焦控制方法、装置、电子系统及存储介质
TWI535285B (zh) Conference system, surveillance system, image processing device, image processing method and image processing program, etc.
CN108717704B (zh) 基于鱼眼图像的目标跟踪方法、计算机装置及计算机可读存储介质
CN106981078B (zh) 视线校正方法、装置、智能会议终端及存储介质
CN108958469B (zh) 一种基于增强现实的在虚拟世界增加超链接的方法
JPH11261868A (ja) 魚眼レンズカメラ装置及びその画像歪み補正方法及び画像抽出方法
TWI742481B (zh) 視訊會議全景影像展開方法
CN107517360B (zh) 一种图像区域遮蔽方法及装置
CN110099220B (zh) 一种全景拼接方法及装置
CN111062234A (zh) 一种监控方法、智能终端及计算机可读存储介质
CN110675350B (zh) 云台相机视场坐标映射方法、装置、存储介质及云台相机
CN112907617A (zh) 一种视频处理方法及其装置
TWI615808B (zh) 全景即時影像處理方法
US11657477B2 (en) Image processing device, image processing system, imaging device, image processing method, and recording medium storing program code
US20220230275A1 (en) Imaging system, image processing apparatus, imaging device, and recording medium
WO2024001342A1 (zh) 成像畸变矫正方法及装置
CN110533577B (zh) 鱼眼图像校正方法及装置
CN115514887A (zh) 视频采集的控制方法、装置、计算机设备和存储介质
CN117395518A (zh) 成像畸变矫正方法及装置
WO2024119902A1 (zh) 图像拼接方法及装置
JP2017010555A (ja) 画像処理方法及び画像処理装置
CN112801873B (zh) 一种全景图像拼接建模的方法及装置
US20240119676A1 (en) Image generation method, apparatus, and system, and computer-readable storage medium
CN108805804B (zh) 一种液晶电视中全景图片处理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23829533

Country of ref document: EP

Kind code of ref document: A1