WO2022040988A1 - Procédé et appareil de traitement d'images, et plateforme mobile - Google Patents

Procédé et appareil de traitement d'images, et plateforme mobile Download PDF

Info

Publication number
WO2022040988A1
WO2022040988A1 PCT/CN2020/111450 CN2020111450W WO2022040988A1 WO 2022040988 A1 WO2022040988 A1 WO 2022040988A1 CN 2020111450 W CN2020111450 W CN 2020111450W WO 2022040988 A1 WO2022040988 A1 WO 2022040988A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
pixel block
pixel
camera
pose
Prior art date
Application number
PCT/CN2020/111450
Other languages
English (en)
Chinese (zh)
Inventor
周游
刘洁
陈希
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to PCT/CN2020/111450 priority Critical patent/WO2022040988A1/fr
Priority to CN202080039127.8A priority patent/CN113950705A/zh
Publication of WO2022040988A1 publication Critical patent/WO2022040988A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10052—Images from lightfield camera

Definitions

  • the present application relates to the technical field of image processing, and in particular, to an image processing method, device and movable platform.
  • non-target subjects In the process of shooting images or videos, users are usually faced with a scene where non-target subjects appear within the shooting angle of view, and these non-target subjects are also captured in the final image or video, which affects the final shooting effect. For example, assuming that the user is photographing a building, there may be some non-target subjects such as passers-by, vehicles, trash cans or telephone poles next to the building. These non-target subjects will block the photographed building, or appear in the In the captured image, it affects the display effect of the image. In order to improve the shooting effect of images or videos to better meet the shooting needs of users, it is necessary to propose a solution for removing non-target shooting objects in images.
  • the present application provides an image processing method, device and movable platform.
  • an image processing method comprising:
  • the second image includes a second pixel block corresponding to a target object, and the target object is the object to be filtered out in the first image occluded object;
  • the first pixel block in the first image is replaced by the second pixel block to generate a replaced first image.
  • an image processing method comprising:
  • the pixel block of the corresponding pixel position is the third image of the static area
  • the first pixel block in the first image is replaced with the pixel block at the corresponding pixel position in the third image.
  • an image processing apparatus includes a processor, a memory, a computer program stored in the memory and executable by the processor, and the processor executes the computer program , implement the following steps:
  • the second image includes a second pixel block corresponding to a target object, and the target object is the object to be filtered out in the first image occluded object;
  • the first pixel block in the first image is replaced by the second pixel block to generate a replaced first image.
  • an image processing apparatus includes a processor, a memory, a computer program stored in the memory and executable by the processor, and the processor executes the computer program , implement the following steps:
  • the first pixel block in the first image is replaced with the pixel block at the corresponding pixel position in the third image.
  • a movable platform is provided, where the movable platform includes a camera device and any one of the image processing devices in the embodiments of the present application.
  • the second image including the target object occluded by the object to be filtered is used to complete the occluded target object in the first image, so as to eliminate the first image.
  • the objects to be filtered out in the image are not only suitable for filtering out dynamic objects, but also for filtering out static objects.
  • the purpose of automatically filtering out the non-shooting target objects in the image can be realized according to the user's needs, which can improve the quality of the image. Display effect and user experience.
  • FIG. 1 is a schematic diagram of filtering out non-shooting target objects in an image according to an embodiment of the present application.
  • FIG. 2 is a flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 3 is a schematic diagram of determining a first pixel block corresponding to an object to be filtered out according to an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a prompting interface for prompting a user to adjust to a second pose according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of adjusting a camera device to a second pose according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of determining a second pose according to an embodiment of the present application.
  • Fig. 7 is a schematic diagram of determining corresponding pixel regions of an object to be filtered and a target object according to an embodiment of the present application.
  • FIG. 8 is a schematic diagram of determining a second orientation according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of determining whether an image captured by a camera device can be used as a second image according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of determining whether an image captured by a camera device can be used as a second image according to an embodiment of the present application.
  • FIG. 11( a ) is a schematic diagram of filtering out dynamic objects according to an embodiment of the present application.
  • FIG. 11( b ) is a schematic diagram of determining a third image according to an embodiment of the present application.
  • FIG. 12 is a flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 13 is a schematic diagram of an application scenario of an embodiment of the present application.
  • FIG. 14 is a schematic diagram of filtering out dynamic objects according to an embodiment of the present application.
  • FIG. 15 is a schematic diagram of a frame selection of an object to be filtered out and an occluded background area according to an embodiment of the present application.
  • FIG. 16 is a schematic diagram of filtering out static objects according to an embodiment of the present application.
  • FIG. 17 is a schematic diagram of a logical structure of an image processing apparatus according to an embodiment of the present application.
  • non-shooting target objects within the shooting angle of view, and these non-shooting target objects are also captured in the image, which affects the display effect of the image.
  • These non-target objects may be dynamic, such as walking passers-by, vehicles, etc., or static, such as trash cans, telephone poles, buildings, etc.
  • the user wants to photograph the house 11 and its surrounding scenery, but the trash can 12 and passers-by in front of the house are also photographed in the image 13, which seriously affects the visual effect of the image. Therefore, it is necessary to remove these non-shooting target objects in the image, as shown in (b) in FIG. 1 , so that the image has a better shooting effect and improves the user experience.
  • the embodiment of the present application provides an image processing method.
  • the second image is collected from the second pose of the target object occluded by the object to be filtered, so as to use the second image to complement the target object occluded by the object to be filtered in the first image, so as to achieve the purpose of removing the object to be filtered.
  • Figure 2 the flow chart of the method is shown in Figure 2, which includes the following steps:
  • S204 Acquire a second image collected by the camera in the second pose, the second image includes a second pixel block corresponding to a target object, and the target object is the target object in the first image that is to be filtered Except for objects occluded by objects;
  • the image processing method in this embodiment of the present application may be performed by a camera device that captures the first image and the second image, and the camera device may be any device with an image or video capture function.
  • the camera device may be a camera, or a device Terminals such as cameras, mobile phones, tablets, laptops, and smart wearable devices with cameras can also be mobile platforms such as drones, unmanned vehicles, and handheld PTZs with cameras.
  • the image processing methods of the embodiments of the present application may also be performed by other devices that are communicatively connected to the camera, for example, a cloud server, and the camera will send the first image and the second image after collecting the first image and the second image.
  • the cloud server of course, if the camera device is a movable platform, the image processing method in this embodiment of the present application may also be executed by a control terminal communicatively connected to the mobile platform, which is not limited in this application.
  • the object to be filtered out in this embodiment of the present application refers to an object that the user wishes to remove from the first image.
  • the object to be filtered out may be a dynamic object or a static object, and there may be one or more objects to be filtered out.
  • the target object in this embodiment of the present application refers to an object in the first image that is occluded by the object to be filtered out, and there may be one or more target objects.
  • the first pixel block corresponding to the object to be filtered out can be determined in the first image, and then a second image containing the target object occluded by the object to be filtered out is acquired, and the second image is used
  • the second pixel block corresponding to the target object in the first image performs replacement processing on the first pixel block in the first image, so that the object to be filtered out in the first image can be eliminated.
  • the target object in another image is complemented by the image including the target object occluded by the object to be filtered, so as to eliminate the object to be filtered out in the other image by acquiring the images collected by the camera in different poses.
  • This method can automatically filter out the non-shooting target objects of the image according to user needs, which can improve the display effect and user experience of the image.
  • the device can automatically identify the object to be filtered from the first image. For example, some identification rules for the object to be filtered can be preset, and then the object to be filtered can be identified according to these rules. For example, the object to be filtered can be automatically identified. Objects such as trash cans, utility poles, and walking vehicles are identified from the first image as objects to be filtered out, and objects located at the edge of the image can also be identified as objects to be filtered out.
  • the identified object to be filtered out can also be displayed to the user, for example, an image of the object to be filtered out is displayed in a frame on the user interface, and subsequent steps are performed after the user confirms.
  • the object to be filtered out may be determined in the first image according to the user's instruction, for example, the user may click or frame the object to be filtered out in the image.
  • the first pixel block may be a pixel block that only includes the object to be filtered out, or a pixel block that includes the object to be filtered out and its surrounding part of the background.
  • the first pixel block may be a human-shaped outline area 31 including only the person to be filtered out, or may be a rectangular area 32 including the person to be filtered out and its surrounding background area.
  • the user instruction for determining the object to be filtered may include a check box input by the user through a human-computer interaction interface, where the check box is used to frame the object to be filtered out.
  • the user can directly draw a selection frame on the user interface to select the objects to be filtered out.
  • the selection frame drawn by the user can be a rectangular selection frame, a circular selection frame or an irregular shaped selection frame, which can be set according to actual needs.
  • the image area selected by the user through the selection frame may be used as the first pixel block corresponding to the object to be filtered out. Since the selection frame drawn by the user may not be accurate enough, there may be some edge areas of the object to be filtered out that are not The frame is selected, but some background areas are frame-selected instead, and when the first pixel block is replaced, there may be incompletely replaced objects to be filtered out in the replaced image, which causes the image to appear after the image is replaced. In order to avoid this phenomenon, after the user inputs the marquee to select the object to be filtered out, the first image can be subjected to superpixel segmentation processing to obtain multiple image areas, and then the marquee selection is performed according to the multiple image areas. pixel block to adjust.
  • the principle of superpixel segmentation processing is to group pixels, and group adjacent pixels with similar texture, color, brightness and other characteristics into a group as an image area, so that the image can be divided into multiple image areas. Then, the pixel blocks framed in the marquee can be adjusted according to the multiple image areas.
  • it can be implemented by using a currently common algorithm, and details are not described herein again.
  • the ratio of the portion of each image region that falls into the marquee to the image region can be adjusted. to adjust the pixel block selected by the marquee. For example, you can first determine the proportion of the part of each image area that falls into the marquee to the image area. If the proportion is greater than a preset proportion, such as greater than 50%, it is considered that the image area is selected by the marquee, and the image area is selected. If it is smaller than the marquee, it is considered that the image area is not selected by the marquee, and the image area is placed outside the marquee to adjust the pixel block selected by the marquee.
  • a preset proportion such as greater than 50%
  • the selection frame in order to ensure that the first pixel block corresponding to the object to be filtered is within the selection frame as much as possible, after adjusting the pixel block selected by the selection frame in the above-mentioned manner, the selection frame can also be enlarged. Expand the pixel block selected by the marquee to serve as the first pixel block.
  • the camera device can be mounted on a movable platform, and the camera device can be controlled by controlling the movement of the movable platform The movement is adjusted so that the second pose occluded by the object to be filtered can be observed and a second image is acquired.
  • the movable platform can be any electronic device that includes a power component that can drive the movable platform to move.
  • the movable platform can be any one of drones, unmanned vehicles, unmanned ships, or intelligent robots.
  • the camera device can also be mounted on the movable platform through the pan-tilt.
  • the movable platform can be provided with a pan-tilt, and the camera device can be fixed on the pan-tilt, and the camera can be controlled by controlling the movement of the movable platform.
  • the movement of the device can also be controlled to make the camera device and the movable platform move relative to each other by controlling the movement of the PTZ, so as to control the movement of the camera device to adjust to the second pose.
  • the second pose may include a second position and a second orientation.
  • the movable platform In a scene where the camera device is mounted on the movable platform through the gimbal, when the camera device is controlled to move, the movable platform can be controlled to move so as to make the camera move.
  • the camera device is located at the second position, and the orientation of the camera device can be adjusted to the second orientation by controlling the rotation of the pan/tilt.
  • a gimbal can be set on the UAV, and a camera device can be installed on the gimbal.
  • the UAV can be controlled to fly to the second position to reach the first position.
  • the pan/tilt can be controlled to rotate, so that the orientation of the camera device is adjusted to the second orientation.
  • a variety of ways can be used in acquiring the second image. For example, it is possible to first determine the second pose where the target object can be observed, and then directly control the camera to adjust to the second pose to collect the second image.
  • the pose of the camera can also be changed continuously to obtain the Multiple frames of images collected in different poses, and then each time a frame of image is collected, it is determined whether the image includes a complete target object, until an image including the complete target object is obtained, which is taken as the second image.
  • the camera device in order to capture the target object that is occluded by the object to be filtered out, it may be automatically determined that the camera can observe the second pose of the target object, and then the camera is controlled to move to adjust to the second pose. pose to acquire the second image.
  • the camera device can be mounted on movable platforms such as drones and unmanned vehicles. Therefore, the position and attitude of the camera device can be automatically calculated to observe the complete target object, and then automatically control the drone, The unmanned vehicle moves to the corresponding position and collects the second image.
  • the drone can capture the second pose of the target object, and then a prompt message indicating the second pose is sent to the user, so that the user can control the movement of the camera to adjust to the A second pose and a second image is acquired.
  • the second pose can be automatically calculated first, and then a prompt message indicating the second pose is sent to the user through the interactive interface.
  • the prompt message can be: Text information can also be image information.
  • prompt information such as "move 100 meters from the current position to the east” and “move to the right from the current position 50 meters” can be displayed on the interactive interface, so that the user can control the camera according to the prompt information.
  • the device is moved to the corresponding position before shooting.
  • the user uses the movable platform to shoot, the user can control the control terminal corresponding to the movable platform according to the prompt information, and control the movable platform to move to the corresponding position through the control terminal.
  • the second pose includes a second position and a second orientation
  • the prompt information can also be image information.
  • an image that identifies the second position in the second pose can be displayed to the user on the interactive interface, wherein, The second position where the target object can be observed can be an area, so this area can be framed in the image, as shown in FIG. 4 .
  • the rotation angle information corresponding to the second orientation adjusted to the second pose can also be displayed to the user, so that the user can adjust the camera device to capture the second image according to the displayed position information and angle information.
  • the position of the target object that can be photographed is related to the relative position of the object to be filtered and the target object, for example, when the distance between the object to be filtered and the target object is far, moving a small distance can collect the complete target object.
  • the distance between the object to be filtered and the target object is relatively short, it may be necessary to move a long distance to collect an image including the complete target object. Therefore, in some embodiments, when determining the second pose, the location information of the object to be filtered and the location information of the target object may be determined first, and then the location information of the object to be filtered and the location information of the target object may be determined. second pose.
  • the size of the object to be filtered also affects the position where the target object can be completely photographed. For example, if the size of the object to be filtered is large, the target object may have to be moved to a farther position to be completely photographed. Small in size, it may be possible to fully capture the target object by moving a small distance. Therefore, in some embodiments, when the second pose is determined according to the position information of the object to be filtered and the position information of the target object, the size of the object to be filtered may be determined, and then the size of the object to be filtered, the size of the object to be filtered and the size of the object to be filtered may be determined. The second pose is determined by dividing the position information of the object and the position information of the target object.
  • the trash can 51 in the figure is the object to be filtered out
  • the house 52 is the target object to be occluded
  • the black dot in the figure represents the position of the camera device.
  • the camera can Bypass the object 51 to be filtered (for example, go behind the object to be filtered, as shown in "Position 2") to capture the complete target object 52, of course, you can also move a distance along the current shooting position to reach "Position 3" so that the target object 52 can fall within the field of view of the camera.
  • the first pose includes a first position and a first orientation
  • the second pose includes a second position and a second orientation
  • the second position passes through the first position and is parallel to a plane where the object to be filtered is located
  • the second orientation points to the position of the object to be filtered, that is, you can translate a distance along the current first position of the camera device to reach the second position, and then adjust the orientation of the camera device to point to the object to be filtered. remove objects.
  • the moving distance when determining the second position, may be determined according to the position information of the object to be filtered, the position information of the target object, and the size of the object to be filtered, and then the moving distance may be determined according to the first position and the size of the object to be filtered.
  • the moving distance determines the second position. For example, as shown in FIG. 6, the small cuboid 61 in the figure is the object to be filtered, the width of the object to be filtered is L, the distance between the object to be filtered and the camera is d1, and the large cuboid 62 is blocked.
  • the target object, the distance between the target object and the camera device is d2, and the object to be filtered and the target object are converted to a view from a top-down perspective.
  • the object to be filtered is shown as 65 in the figure
  • the area of the target object that is occluded by the object to be filtered is shown as 64 in the figure
  • the "position A" in the figure is the first position
  • the schematic diagram of the image plane 66 is the camera
  • the image plane schematic diagram 67 is a schematic diagram of the image collected by the camera at "position B", where "position B" is the position where the camera device can just observe the left edge of the occluded area of the target object.
  • position, "position B” can be reached by translation distance D from the first position, and it can be seen from Figure 6 that the moving distance D can be solved by formula (1):
  • the distance d1 between the object to be filtered and the camera device and the distance d2 between the target object and the camera device may be determined by using multiple images collected by the camera device in different poses.
  • the width L of the object to be filtered out can be determined according to the distance between the object to be filtered out and the camera device and the imaging size of the object to be filtered out.
  • the second location can be any location in this area.
  • the three-dimensional space coordinates of "position B" can be determined according to the current three-dimensional space coordinates of the first position and the moving distance D, the three-dimensional space coordinates corresponding to the second position can be further determined, and the camera is controlled to move to the second position. Location.
  • an area including the object to be filtered out is determined, for example, as shown in FIG. 7 , an area 71 including the object to be filtered out 70 is determined, and then an annular area 72 surrounding the area 71 is determined.
  • a plurality of feature points can be extracted from the area 71 first, and the feature point extraction can be performed by using an existing feature point extraction algorithm, which will not be repeated here.
  • the matching points of the extracted feature points in the remaining multi-frame images can be determined, and then the optical flow vector of each feature point can be determined according to the matching points of these feature points in the remaining images, and the optical flow vector of the feature points can be fitted according to the optical flow vector of the feature points to be filtered.
  • the center of the object that is, the area 71
  • the center of the object is relative to the optical flow vector of each image, so that the matching points of the center of the object to be filtered (that is, the area 71) in the remaining multi-frame images can be determined, according to the center of the object to be filtered out.
  • the BA (Bundle Adjustment) algorithm can be used to determine the internal and external parameters of the camera, and the depth distance from the center of the object to be filtered is determined according to the internal and external parameters of the camera, which is the distance between the object to be filtered and the camera.
  • distance d1 For the distance d2 between the target object and the camera device, feature points can be extracted from the annular region 72, and then a similar method is used to determine the distance d2 between the target object and the camera device, which will not be repeated here.
  • the second orientation when the second orientation is determined, the second orientation may be determined according to the first position and the position of the object to be filtered out in the image frame captured by the camera. For example, during the moving process of the camera device, it can detect the position of the object to be filtered out in the captured image in real time, and can continuously adjust the orientation of the camera device to keep the object to be filtered out in the center of the image screen, so that when the camera device When moving to the second position, the second orientation can also be determined.
  • the center of the object to be filtered out in the first image on the image screen and the pose parameters corresponding to the first pose, it can be determined that when the camera moves to the second position, the center of the object to be filtered out should be located at the center of the screen at the second position.
  • the second orientation corresponds to the attitude angle, thereby determining the second orientation.
  • the second orientation when determining the second orientation, may also be determined according to the first position, the positions of the left and right endpoints of the object to be filtered, and the positions of the left and right endpoints of the target object. For example, as shown in FIG. 8 , which side of the first position the second position is located on can be determined according to the three-dimensional coordinates of the first position and the second position, and when the second position is located on the right side of the first position, it can be determined according to the three-dimensional coordinates of the first position and the second position.
  • the left endpoint A of the object to be filtered and the right endpoint D of the target object determine a connecting line AD
  • the second orientation is to point to the object to be filtered along the connecting line AD.
  • the attitude angle corresponding to the line connecting the two endpoints can also be solved.
  • a connecting line BC can be determined according to the left endpoint B of the object to be filtered and the right endpoint C of the target object, and the second orientation is along the connecting line BC pointing to the object to be filtered. Filter out objects.
  • the attitude angle corresponding to the connection line BC can be determined according to the three-dimensional coordinates of the left endpoint B of the object to be filtered and the right endpoint C of the target object.
  • the second image may also be acquired by continuously adjusting the pose of the camera to acquire the image and then judging whether the acquired image can be used as the second image.
  • the pose of the camera can also be continuously changed to obtain multiple frames of images collected by the camera at different poses. Each time the camera collects a frame of image, it can determine whether the image includes the corresponding image of the target object.
  • the second pixel block of if included, the image is regarded as the second image.
  • the determination of whether the image includes the second pixel block may be determined by the user, or may be determined automatically by the device executing the image processing method. Taking the camera device mounted on a mobile platform such as a drone as an example, the user can adjust the posture of the drone and collect images in different postures, and then the drone can send the collected images back to the control terminal, and the user can determine the image. When the target object is included in the , you can click on the image to use it as a second image. Of course, it is also possible to automatically determine whether the collected image can be used as the second image by performing certain processing and identification on the collected image.
  • a plurality of first feature points may be extracted from the first pixel block, and A plurality of second feature points are extracted from the surrounding area of the first pixel block, and for each frame of image collected after the camera device changes the pose, the first matching point and the second feature of the first feature point in the image can be determined point the second matching point in the image, and then determine whether the image includes the second pixel block according to the positional relationship between the first matching point and the second matching point in the image.
  • the first feature point may be a feature point located within the first side of the first pixel block
  • the second feature point may be a feature point located outside the second side of the first pixel block
  • the first side is the opposite side of the second side. For example, the first side is the left side of the first pixel block, the second side is the right side of the first pixel block, and the first side is the first side.
  • FIG. 9 is a schematic diagram of the first image 90 , the first pixel block 91 can be determined from the first image 90 , and the first pixel block 91 is on the first side of the first pixel block 91 .
  • a plurality of first feature points 92 are extracted within (ie, the left side), and a plurality of second feature points 93 are extracted outside the second side (ie, the right side) of the first pixel block.
  • FIG. (b) is a schematic diagram of the image 94
  • the first feature point is at the first matching point 95 of the image 94 and the second feature point is at the second matching point 96 of the image 94
  • the second matching point 96 is located on the first side (left side) of the first matching point 95, and it is considered that the target object is not at this time. is blocked by the object to be filtered out, so it can be determined that the image includes the second pixel block corresponding to the target object.
  • the plurality of second feature points may be located in a ring-shaped pixel block surrounding the first pixel area, at positions according to the first matching point and the second matching point
  • the preset number may be determined according to actual requirements, for example, 90% of the second matching points may be located on one side of the first matching point. As shown in FIG.
  • (a) is a schematic diagram of the first image 100
  • the first pixel block 101 can be determined from the first image 100
  • a plurality of first feature points 102 can be extracted from the first pixel block 101 , extract a plurality of second feature points 104 in the annular pixel block 103 around the first pixel block 101, when the camera changes the pose to collect a frame of image
  • (b) is a schematic diagram of the image 105
  • the first feature point 102 is at the first matching point 106 of the image 105
  • the second feature point 104 is at the second matching point 107 of the image 105
  • the positional relationship between the first matching point 106 and the second matching point 107 can be determined , as shown in the figure, when more than a certain number of second matching points 107 (for example, more than 90% of the total number of second matching points) are located on one side of the first matching point 106 , it is considered that the target object has not been filtered at this time.
  • a certain number of second matching points 107 for
  • the target object occluded by the object to be filtered may not be completely captured.
  • the A complete target object is captured by adjusting the pose of the camera device to complement the target object occluded by the object to be filtered out in the first image.
  • prompt information may be sent to the user to prompt the user that the object to be filtered out in the first image cannot be filtered out in the current scene.
  • a prompt message indicating that the object to be filtered cannot be filtered is sent to the user, wherein the prompt message can be displayed in the form of a pop-up window, for example, it can be displayed in the user
  • the interactive interface displays pop-up information, prompting the user that the currently selected object to be filtered cannot be filtered out.
  • the first preset condition may be at least one of the following: the first distance between the object to be filtered and the target object is less than a first preset threshold, or the second distance between the target object and the camera is less than the first distance Two preset thresholds, or the distance relationship between the first distance and the second distance does not satisfy the preset second condition.
  • the first preset threshold, the second preset threshold and the second condition can be flexibly set according to the actual scene. For example, if the distance between the object to be filtered and the target object is less than 1 meter, the complete target object cannot be photographed.
  • the first preset threshold is set to 1 meter.
  • the second preset threshold and the second preset condition may be set by similar means, and details are not described herein again.
  • the second pixel block before performing the replacement process on the first pixel block of the first image with the second pixel block in the second image, the second pixel block may be determined in the collected second image first.
  • the distance between the pixel points of the first image and the pixel points of the second image may be determined first.
  • the mapping area of the first pixel block in the second image is determined as the second pixel block, and then the second pixel block is used to replace the first pixel block in the first image.
  • a third feature point may be extracted from the peripheral area of the first pixel block in the first image, and A third matching point of the third feature point is determined in the second image, and then the mapping relationship is determined according to the third feature point and the third matching point.
  • the mapping relationship between the pixels of the first image and the pixels of the second image can be represented by a homography matrix. For example, assuming that the pixel coordinates of the pixels on the first image are P1, The pixel coordinate of the pixel point is P2, and the homography matrix is H, then the pixel point of the first image and the pixel point of the second image satisfy the formula (2):
  • the homography matrix H Since the homography matrix H has 8 unknowns, it needs at least 4 pairs of feature points and matching points to solve. Therefore, at least 4 third feature points can be extracted from the surrounding area of the first pixel block in the first image (such as the area surrounding the first pixel block), and then it is determined that these at least 4 pixel points are in the second image. The third matching point of , solve H according to the third feature point and the third matching point.
  • the RANSAC Random sample consensus
  • the RANSAC Random sample consensus algorithm
  • the mapping area of the first pixel block in the first image in the second image can be determined according to H, and then the mapping area is used as the second pixel block to replace the first pixel area in the first image block to complement the target object in the first image that is occluded by the object to be filtered out.
  • a ring-shaped pixel block surrounding the first pixel block can also be determined in the first image, and then determined in the second image.
  • a matching ring-shaped block matched with the ring-shaped pixel block, and then the pixel block surrounded by the matching ring-shaped block in the second image is used as the second pixel block.
  • the first image can be an image with a better shooting effect.
  • the camera device may collect multiple images in the first pose, and then select the first image from the multiple images, wherein the first image may be selected by the user, or the image processing method may be executed by the user. The device is automatically selected according to the image clarity, brightness, picture composition and other information.
  • the category information of the object to be filtered out may be determined first, and the category information is used for The object to be filtered is identified as a dynamic object or a static object, and a corresponding processing method is adopted to eliminate the object to be filtered according to the category information of the object to be filtered.
  • determining whether the object to be filtered out is a dynamic object or a static object may be determined according to a plurality of images collected by the camera device in the first pose.
  • each pixel of the object to be filtered out in the first image may be relative to the plurality of images
  • the object to be filtered out is a static object
  • the first pixel block corresponding to the object to be filtered out is determined in the first image, and the image captured by the camera in the second pose including the second pixel is obtained.
  • the second image of the block, and the second pixel block is used to replace the first pixel block in the first image, so as to generate a step of replacing the processed first image.
  • the third pixel block corresponding to the object to be filtered out in the first image may be determined, and then the third pixel block corresponding to the object to be filtered out in the first image may be determined, and then the third pixel block acquired by the camera in the first pose is determined.
  • the fourth pixel block located at the pixel position corresponding to the pixel position of the third pixel block in other images other than one image it is determined from the other images that the fourth pixel block is the third image of the static area (that is, the third image The corresponding area of the three-pixel block is not blocked), and the fourth pixel block in the third image is used to replace the third pixel block in the first image.
  • the third pixel block 110 corresponding to the object to be filtered can be determined in the first image, and then it is determined that the pixel position of the third pixel block 110 is located in another image (such as image 1 in the figure) , image 2, image 3) in the fourth pixel block (111 in the figure) corresponding to the pixel position, then it can be determined whether the fourth pixel block 111 is a static area, if so, then this image is used as the third image
  • the fourth pixel block 111 in the image 1 is a static area
  • the image 1 is regarded as the third image
  • the fourth pixel block 111 in the image 1 is used to replace the third pixel block 110 in the first image.
  • the dynamic area in the other images may be determined first, and then the first image is determined for the first image.
  • the pixel block whose pixel position of the third pixel block is located in the corresponding pixel position in the other images can be determined according to the order of acquisition of the other images and the first image from near to far, until The pixel block corresponding to the pixel position does not overlap with the dynamic area (ie, is not blocked), and the other image is regarded as the third image.
  • the first image is the Kth frame image collected by the camera
  • the other images are the K+1th frame, the K+2th frame, and the K+th frame collected by the camera respectively.
  • 3 frames, etc. you can first determine the dynamic area in the K+1th frame, the K+2th frame, and the K+3th frame.
  • the optical flow vector of multiple frames of images If the modulo length of the optical flow vector of a pixel point is greater than a certain threshold, the pixel point is considered to be moving, and then the pixels determined to be moving are clustered to obtain multiple pixel point sets.
  • the area where the number of pixels in the set is greater than a certain value is considered as a motion area.
  • the rectangular area and the circular area in the image are dynamic areas, and the remaining areas are static areas, determine the third pixel block 121 corresponding to the object to be filtered in the first image, and the pixel position 121 where the third pixel block is located is in the K+th
  • the pixel block of the corresponding pixel position in the 1st frame, the K+2th frame, and the K+3th frame is the area 122 framed by the dotted line.
  • the pixel block 122 corresponding to the pixel position of the third pixel block 121 in a frame of image overlaps with the dynamic region in the K+1th frame, and if so, then determine the Kth Whether the pixel block 122 corresponding to the pixel position of the third pixel block 121 in the +2 frames overlaps with the dynamic area, when it is determined that the K+2th frame meets the requirements, the K+2th frame is used as the third image, The third pixel block of the first image is replaced with the pixel block corresponding to the pixel position in this frame.
  • the camera device When the camera device collects images in the first attitude, it usually collects multiple frames of images continuously to form an image sequence. The position of the image does not change much, so it is not suitable to use adjacent frames to filter out the dynamic objects of the first image. If these image frames are judged one by one, it is more resource-intensive. Therefore, when acquiring multiple images collected by the camera, some images that can reflect the changes of dynamic objects can be selected from the image sequence collected by the camera, so that these images can be used more efficiently to filter out dynamic objects. Therefore, in some embodiments, other images in the plurality of images except the first image may be images whose differences from the first image exceed a preset threshold, or images that are separated from the first image by a specified frame. For example, the first image can be used as a reference.
  • the image is acquired as one of the above-mentioned multiple images. For example, if the first image is the 5th frame of the image sequence, the other images are the 10th, 15th, 20th, and 25th frames in sequence. Wait.
  • the dynamic objects in the image when the static objects in the image are filtered out by collecting images of different poses, the dynamic objects in the image will interfere with the filtering out of the static objects to a certain extent, resulting in an inability to filter out the static objects well.
  • Static objects therefore, in some embodiments, before using the above method to filter out static objects, dynamic objects in the image may be filtered out first.
  • the present application also provides an image processing method, which can be used to automatically remove dynamic objects in an image.
  • the method is shown in FIG. 12 and includes the following steps:
  • the image processing method in this embodiment of the present application may be performed by a camera device that collects the first image and the second image
  • the camera device may be any device with an image acquisition function.
  • the camera device may be a camera, or a camera equipped with a camera.
  • Terminals such as cameras, mobile phones, tablets, laptops, smart wearable devices, etc., can also be mobile platforms such as drones and unmanned vehicles with cameras.
  • the image processing method of the present application can also be executed by other devices that are communicatively connected to the camera, for example, a cloud server, and the camera collects the first image and the second image and sends them to the cloud Server processing, of course, if the camera device is a movable platform, the image processing method of the present application can also be executed by a control terminal communicatively connected to the movable platform, which is not limited in the present application.
  • the dynamic objects to be filtered out in the embodiments of the present application are objects that the user wishes to remove from the image.
  • the dynamic objects to be filtered out may be determined by the user or selected by the user, and there may be one or more dynamic objects to be filtered out.
  • the first image can be selected from the image sequence continuously collected by the camera in a certain fixed pose, wherein the first image can be selected by the user, or can be automatically selected by the device executing the image processing method, such as automatically selecting from the image sequence. Select an image with better clarity, composition or shooting angle as the first image.
  • the first image After determining the first image, it is possible to determine the first pixel block corresponding to the dynamic object to be filtered out in the first image, then determine multiple frames of second images from the image sequence, and determine the first pixel block from the second image
  • the pixel block corresponding to the pixel position at the pixel position is the third image of the static area (that is, the image in which the pixel block corresponding to the pixel position is not occluded), and then the corresponding pixel position of the first pixel block in the third image is used.
  • the pixel block replaces the first pixel block in the first image to remove dynamic objects.
  • the adjacent frames of the first image may not be used to filter out the dynamic object to be filtered out.
  • some images that are quite different from the first image can be selected from the image sequence as
  • the second image for example, the second image may be an image whose difference from the first image exceeds a preset threshold, or an image that is spaced apart from the first image by a specified frame.
  • the first image can be used as a reference.
  • the image is acquired as the second image, It is also possible to acquire images with a specified frame interval from the first image. For example, if the first image is the 5th frame of the image sequence, the second image is the 10th frame, the 15th frame, the 20th frame, and the 25th frame in sequence.
  • the following operations may be performed for each frame of the second image: calculating the light intensity of each pixel of the first image relative to the second image
  • the flow vector is determined from each pixel point of the first image, and the target pixel point whose modulo length of the optical flow vector is greater than the preset threshold is subjected to clustering processing to obtain the first pixel block corresponding to the dynamic object.
  • the optical flow vector of each pixel of the first image and its adjacent one or more frames of images can be calculated.
  • the modulo length of the optical flow vector of the pixel is greater than a certain threshold, it is considered that the pixel is moving, and then it is determined as The moving pixels are clustered to obtain multiple sets of pixels, and the area where the number of pixels in the set is greater than a certain value (the number may be too small and the noise can be ignored) is considered to be a dynamic object.
  • the dynamic regions in the plurality of second images can be determined, and for each For the first pixel block, the corresponding third image can be determined in the following manner, and the pixel position of the dynamic object of the first image can be determined in the second image in the order of the second image and the first image acquisition sequence from near to far. until the pixel block corresponding to the pixel position does not overlap the dynamic area, the second image is regarded as the third image.
  • a reference image (ie, the first image) can be determined from multiple frames of images collected by the camera in the same pose, and then the first image corresponding to the dynamic object to be filtered can be determined from the reference image.
  • Pixel block by determining whether the pixel block corresponding to the pixel position of the first pixel block in other images is a static area, the pixel area corresponding to the pixel position of the first pixel block can be quickly screened out. block the unoccluded image, and then replace the first pixel block in the first image with the pixel block corresponding to the pixel position of the image, which can quickly and efficiently remove the dynamic object in the first image.
  • the user can use the control terminal 132 to control the camera device 133 mounted on the drone 131 to collect images, and the drone 131 can capture images.
  • the images collected by the camera 133 are sent back to the control terminal so as to be displayed to the user. Filtering out the objects in the image can be performed by the control terminal.
  • the following describes the filtering methods of dynamic objects and static objects respectively. Since dynamic objects will cause certain interference to the filtering of static objects, the dynamic objects can be filtered out first, and then the dynamic objects can be filtered out. Filter out static objects.
  • the camera can be controlled to collect a series of image sequences at a certain fixed pose, and then the user selects a reference image I0 from the image sequence, or the control terminal automatically selects the reference image I0 from the image sequence.
  • the key frame is an image frame with a large difference from the reference image I0, for example, the angle or displacement of an object may be different from the angle or displacement of the object in the reference image I0
  • the reference image I0 is the Kth frame
  • the key frame can be an image frame with an interval of 5, 10, 15, and 20 from the Kth frame.
  • the remaining key frames also calculate the optical flow vector with the two frames (or multiple frames) before and after itself, respectively, and determine the dynamic area and the static area of each key frame.
  • each dynamic object of the k0th frame is a static area in the corresponding area of the frame.
  • the triangle, square and circular areas in the figure represent dynamic areas, and the rest of the areas are static areas.
  • the corresponding area of such as the circular dotted area in the figure, is a static area, so the k-1th frame can be used to fill the circular dynamic area of the k0th frame.
  • the remaining triangular dynamic areas and square dynamic areas need to be filled by the k-5th frame and the k7th frame respectively.
  • each dynamic object can be used to replace the dynamic object of the k0 frame in the corresponding area of the key frame, so as to achieve the purpose of filtering out the dynamic object.
  • the static object to be filtered out can be determined in the reference image I0, which can be automatically recognized by the control terminal, or can be determined by the user by frame selection on the interactive interface. As shown in Figure 15, the user can select the static object to be filtered out in the reference image I0 (as shown in Figure (1)). Since the frame selected by the user is not very accurate, the selection frame can be adjusted automatically, such as The benchmark image I0 is subjected to superpixel segmentation to obtain multiple image areas, and then the ratio of the part of each area that falls into the selected frame to the image area is determined. outside the box. The frame adjusted by the above method is shown in Figure (2).
  • feature points can be extracted, and feature point tracking and matching between multiple frames of images, as well as feature point tracking and matching of the previous and subsequent frames, can be used to determine the static object. and the depth distance of the background area. details as follows:
  • feature point extraction is performed on the corresponding area of the static object on the reference image.
  • the feature point extraction can use a general algorithm, such as Harris algorithm, SIFT algorithm, SURF algorithm, ORB algorithm, etc.
  • the sparse method can be used to first extract the feature points of the image.
  • the corner points can be selected as the feature points.
  • the optional corner detection algorithms Corner Detection Algorithm are: FAST (features from accelerated segment test ), SUSAN, and Harris operator, etc. The following is an example of using the Harris Corner Detection Algorithms algorithm:
  • matrix A as a construction tensor, such as formula (3)
  • Ix and Iy are the gradient information of a certain point on the image in the x and y directions, respectively, and the function Mc can be defined as the following formula (4):
  • det(A) is the determinant of matrix A
  • trace(A) is the trace of matrix A
  • ⁇ is the parameter to adjust the sensitivity
  • the set threshold is Mth.
  • the displacement h of the feature point before and after the image frame can be obtained through the iteration of formula (5),
  • the feature points can be updated continuously.
  • the center of static objects does not necessarily have feature points. Therefore, for the center of the static object, it is necessary to use the fitted optical flow vector to determine the position of the center of the static object in each image, so that the BA algorithm can be used to obtain the three-dimensional coordinates of the center of the static object.
  • the center point of the static object can be estimated from the optical flow vectors of other feature points within the corresponding region of the static object framed in the image. Specifically as formula (6):
  • x i is the optical flow vector of the feature points in the frame
  • w i is the weight, which can be determined according to the 2D image position of the feature point and the center point, as shown in formula (7):
  • ⁇ is adjusted according to experience and is an adjustable parameter
  • d i represents the distance from the feature point i to the midline point (u i , v i ) represents the 2D image pixel coordinates of the feature point i
  • (u 0 , v 0 ) represents the 2D image pixel coordinates of the center point of the target frame.
  • the disparity and optical flow of the center of the static object can be calculated, and the three-dimensional depth information of the center of the static object can be obtained.
  • the 3D depth information of the background region can be calculated.
  • d1 and d2 are the depth distance of the static object obtained in one step and the depth distance of the background area.
  • the maximum width L of the static object can be determined according to the size of the static object in the image and the depth distance of the static object.
  • Figure 6 shows that the drone is flying to the right, so the limit position where all the occluded background areas can be seen is that the left edge of the occluded background area is just observed. Of course, the drone can also fly to the left, and the corresponding limit viewing angle is to just see the right edge of the occluded background area.
  • the camera orientation can be adjusted at the same time, and the right edge of the pixel area corresponding to the object to be filtered can be centered. Through the above method, the adjustment of the camera pose can be completed, and the purpose of adjusting the viewing angle is achieved.
  • the pose where the occluded background area can be observed is determined, and the drone can be automatically controlled to adjust to the pose, and the image In can be obtained by shooting at this pose.
  • Feature points can be extracted from the pixel region corresponding to the background region in the reference image I0, and feature point matching can be performed in the image In to obtain a matching point queue.
  • the H matrix represents the mapping relationship between two matched pixels on two images collected by the camera at different poses, as shown in formula (8):
  • x 0 is a feature point on the background region of image I0
  • x n is the point in image In that matches x 0 .
  • H matrix to represent the mapping relationship between two points actually requires these pixels to be on the same plane in space.
  • the background area can be treated as a plane. Therefore, when d1 is relatively small (for example, less than 100m), the user should also be reminded that the filtering effect is poor or cannot be filtered.
  • the tolerance parameter that is, the maximum degree of unevenness used in fitting the plane can be selected according to the depth of the background. (such as removing 2% of the background depth), if the plane cannot be fitted, the user will be prompted that the filtering effect is poor or cannot be filtered.
  • the H matrix has 8 unknowns and requires at least 4 pairs of points to calculate.
  • the RANSAC (Random sample consensus) algorithm can be used to effectively filter out feature points and matching points with poor matching degree, further improve the effectiveness of the results, and obtain a more accurate H matrix.
  • the image In can be all projected onto the camera pose (camera pose) of the camera corresponding to the image I0 according to the H matrix to obtain the image In', and then the The pixel area corresponding to the object to be filtered out in the image In' is replaced by the area corresponding to the coverage image I0, and the static object can be removed.
  • the present application also provides an image processing apparatus.
  • the image processing apparatus includes a processor 171 , a memory 172 , and a computer program stored in the memory 172 and executable by the processor 171 .
  • the processor executes the computer program, the following steps are implemented:
  • the second image includes a second pixel block corresponding to a target object, and the target object is the object to be filtered out in the first image occluded object;
  • the first pixel block in the first image is replaced by the second pixel block to generate a replaced first image.
  • the processor when the processor is configured to acquire the second image captured by the camera in the second pose, the processor is specifically configured to:
  • the camera is controlled to move to adjust to the second pose and capture the second image.
  • the processor when the processor is configured to acquire the second image captured by the camera in the second pose, the processor is specifically configured to:
  • the processor when the processor is configured to determine the second pose, it is specifically configured to:
  • the second pose is determined according to the position information of the object to be filtered and the position information of the target object.
  • the processor is configured to determine the second pose according to the position information of the object to be filtered and the position information of the target object, and is specifically configured to:
  • the second pose is determined according to the position information of the object to be filtered, the position information of the target object, and the size of the object to be filtered.
  • the first pose includes a first position and a first orientation
  • the second pose includes a second position and a second orientation
  • the second position is located on a straight line passing through the first position and parallel to the plane where the object to be filtered is located, and the second orientation points to the position where the object to be filtered is located.
  • the second location is determined by:
  • the second position is determined according to the first position and the moving distance.
  • the second orientation is determined by:
  • the second orientation is determined according to the first position, the positions of the left and right endpoints of the object to be filtered, and the positions of the left and right endpoints of the target object.
  • the second posture includes a second position and a second orientation
  • the processor is configured to send prompt information indicating the second posture to the user, when, specifically:
  • the image marked with the second position is displayed to the user, and the rotation angle information corresponding to the second orientation is displayed.
  • the processor when the processor is configured to acquire the second image captured by the camera in the second pose, the processor is specifically configured to:
  • An image including the second pixel block is used as the second image.
  • the processor when the processor is configured to determine whether the image includes the second pixel block, it is specifically configured to:
  • Whether the image includes the second pixel block is determined according to the positional relationship between the first matching point and the second matching point in the image.
  • the first feature point is located within a first side of the first pixel block, and the second feature point is located outside a second side of the first pixel block, wherein the the first side is the opposite side of the second side;
  • the processor When the processor is used to determine whether the image includes the second pixel block according to the positional relationship between the first matching point and the second matching point, it is specifically used for:
  • a plurality of the second feature points are located in a ring-shaped pixel block surrounding the first pixel area, and the processor is configured to determine the first matching point and the second matching point according to the first matching point and the second matching point.
  • the processor is specifically used for:
  • the camera device is mounted on a movable platform, and the processor is used to control the movement of the camera device, specifically:
  • Movement of the movable platform is controlled to control movement of the camera.
  • the camera device is mounted on a movable platform through a pan/tilt
  • the processor is used to control the movement of the camera device, specifically:
  • the movable platform is controlled to move, and/or the pan/tilt is controlled to generate relative motion between the camera and the movable platform, so as to control the camera to move.
  • the camera device is mounted on the movable platform through a pan/tilt head
  • the second pose includes a second position and a second orientation
  • the processor is configured to control the camera device to move so as to make the camera device move.
  • the camera device is adjusted to the second pose, it is specifically used for:
  • the movable platform is controlled to move so that the camera is located at the second position; and the pan-tilt is controlled to rotate so that the orientation of the camera is adjusted to the second orientation.
  • the movable platform includes any one of an unmanned aerial vehicle, an unmanned vehicle, and an unmanned boat.
  • the processor when the processor is configured to determine the first pixel block corresponding to the object to be filtered out in the first image, it is specifically configured to:
  • the first pixel block corresponding to the object to be filtered out is determined from the first image.
  • the instruction includes a check box input by the user through a human-computer interaction interface, and the check box is used to frame the static target object.
  • the first pixel block is a pixel block selected by the marquee, and the device is further configured to:
  • the processor when the processor adjusts the pixel blocks selected by the frame based on the plurality of image regions, the processor is specifically configured to:
  • the pixel block selected by the frame is adjusted according to the ratio of the portion of each image area that falls within the frame and each image area in the plurality of image areas.
  • the apparatus is also used to:
  • the preset first condition includes one or more of the following:
  • the first distance between the object to be filtered and the target object is less than a first preset threshold
  • the distance magnitude relationship between the first distance and the second distance does not satisfy a preset second condition.
  • the apparatus is also used to:
  • the second pixel block is determined in the second image.
  • the processor when the processor is configured to determine the second pixel block in the second image, it is specifically configured to:
  • the mapping area of the first pixel block in the second image is determined according to the mapping relationship as the second pixel block.
  • the processor when the processor is configured to determine the mapping relationship between the pixels of the first image and the pixels of the second image, it is specifically configured to:
  • the mapping relationship is determined based on the third feature point and the third matching point.
  • the processor when the processor is configured to determine the second pixel block in the second image, it is specifically configured to:
  • a pixel block surrounded by the matching ring-shaped block in the second image is used as the second pixel block.
  • the processor when the processor is configured to acquire the first image captured by the camera in the first pose, the processor is specifically configured to:
  • the first image is determined among the plurality of images.
  • the processor before the processor is configured to determine the first pixel block corresponding to the object to be filtered out in the first image, the processor is further configured to:
  • the determining of the first pixel block corresponding to the object to be filtered out in the first image is performed, and the acquisition of the image data collected by the camera in the second pose including the second pixel block is performed.
  • the second image of the pixel block, and the step of replacing the first pixel block in the first image with the second pixel block to generate the replaced first image is performed.
  • the apparatus is also used to:
  • the object to be filtered is a dynamic object, perform the following steps:
  • the third pixel block in the first image is replaced with the fourth pixel block in the third image.
  • the processor when the processor is configured to determine the category of the object to be filtered out from the plurality of images, the processor is specifically configured to:
  • the category information of the object to be filtered out is determined according to the optical flow of each pixel of the object to be filtered out relative to other images in the plurality of images except the first image.
  • the processor when the processor is configured to determine from the other images that the fourth pixel block is the third image of the static area, the processor is specifically configured to:
  • the pixel block of the corresponding pixel position in the other image where the pixel position of the third pixel block is located determines the pixel block of the corresponding pixel position in the other image where the pixel position of the third pixel block is located, until the corresponding pixel position If the pixel block of the pixel position does not overlap with the dynamic area, the other image is used as the third image.
  • the other difference from the first image exceeds a preset threshold
  • the other images are images spaced apart from the first image by a specified frame.
  • the present application also provides another image processing apparatus, the image processing apparatus includes a processor, a memory, a computer program stored in the memory and executable by the processor, and the processor executes the computer program , implement the following steps:
  • the first pixel block in the first image is replaced with a pixel block at the corresponding pixel position in the third image.
  • the processor when used to determine the first pixel block corresponding to the dynamic object in the first image, it is specifically used to:
  • the target pixel points are clustered to obtain the first pixel block corresponding to the dynamic object.
  • the processor is configured to determine from the plurality of second images that the pixel block corresponding to the pixel position is the third image of the static area; specifically:
  • the second image and the first image acquisition sequence from near to far determine the pixel block where the pixel position of the first pixel block is located in the corresponding pixel position in the second image, until If the pixel block corresponding to the pixel position does not overlap with the dynamic area, the second image is used as the third image.
  • the difference between the second image and the first image exceeds a preset threshold
  • the second image is an image spaced apart from the first image by a specified frame.
  • the present application also provides a movable platform, and the movable platform can be any device such as an unmanned aerial vehicle, an unmanned vehicle, an unmanned ship, an intelligent robot, and a handheld PTZ.
  • the movable platform includes a camera device and an image processing device.
  • the image processing device can implement any of the image processing methods in the embodiments of the present application. For specific implementation details, refer to the descriptions of the embodiments in the above image processing methods. No longer.
  • an embodiment of the present specification further provides a computer storage medium, where a program is stored in the storage medium, and when the program is executed by a processor, the image processing method in any of the foregoing embodiments is implemented.
  • Embodiments of the present specification may take the form of a computer program product embodied on one or more storage media having program code embodied therein, including but not limited to disk storage, CD-ROM, optical storage, and the like.
  • Computer-usable storage media includes permanent and non-permanent, removable and non-removable media, and storage of information can be accomplished by any method or technology.
  • Information may be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
  • PRAM phase-change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • ROM read only memory
  • EEPROM Electrically Erasable Programmable Read Only Memory
  • Flash Memory or other memory technology
  • CD-ROM Compact Disc Read Only Memory
  • CD-ROM Compact Disc Read Only Memory
  • DVD Digital Versatile Disc
  • Magnetic tape cassettes magnetic tape magnetic disk storage or other magnetic storage devices or any other non-

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

Procédé et appareil de traitement d'images, et plateforme mobile. Le procédé fait appel aux étapes suivantes : l'acquisition d'une première image collectée par un appareil photographique dans une première pose, et la détermination, dans la première image, d'un premier bloc de pixels correspondant à un objet à filtrer ; l'acquisition d'une seconde image collectée par l'appareil photographique dans une seconde pose, la seconde image comprenant un second bloc de pixels correspondant à un objet cible, et l'objet cible étant un objet, qui est obscurci par l'objet à filtrer, dans la première image ; et la réalisation d'un traitement de remplacement sur le premier bloc de pixels dans la première image au moyen du second bloc de pixels, de façon à générer une première image qui a été soumise à un traitement de remplacement.
PCT/CN2020/111450 2020-08-26 2020-08-26 Procédé et appareil de traitement d'images, et plateforme mobile WO2022040988A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2020/111450 WO2022040988A1 (fr) 2020-08-26 2020-08-26 Procédé et appareil de traitement d'images, et plateforme mobile
CN202080039127.8A CN113950705A (zh) 2020-08-26 2020-08-26 图像处理方法、装置及可移动平台

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/111450 WO2022040988A1 (fr) 2020-08-26 2020-08-26 Procédé et appareil de traitement d'images, et plateforme mobile

Publications (1)

Publication Number Publication Date
WO2022040988A1 true WO2022040988A1 (fr) 2022-03-03

Family

ID=79327044

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/111450 WO2022040988A1 (fr) 2020-08-26 2020-08-26 Procédé et appareil de traitement d'images, et plateforme mobile

Country Status (2)

Country Link
CN (1) CN113950705A (fr)
WO (1) WO2022040988A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118175190A (zh) * 2024-05-13 2024-06-11 国网山东省电力公司阳信县供电公司 用于绝缘杆式杆塔设备的巡检方法、系统、设备及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110103644A1 (en) * 2009-10-30 2011-05-05 Zoran Corporation Method and apparatus for image detection with undesired object removal
CN106056534A (zh) * 2016-05-31 2016-10-26 中国科学院深圳先进技术研究院 基于智能眼镜的遮挡物透视方法及装置
CN106791393A (zh) * 2016-12-20 2017-05-31 维沃移动通信有限公司 一种拍摄方法及移动终端
CN109035185A (zh) * 2018-06-29 2018-12-18 努比亚技术有限公司 一种图像处理方法及终端
CN109167893A (zh) * 2018-10-23 2019-01-08 Oppo广东移动通信有限公司 拍摄图像的处理方法、装置、存储介质及移动终端

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110103644A1 (en) * 2009-10-30 2011-05-05 Zoran Corporation Method and apparatus for image detection with undesired object removal
CN106056534A (zh) * 2016-05-31 2016-10-26 中国科学院深圳先进技术研究院 基于智能眼镜的遮挡物透视方法及装置
CN106791393A (zh) * 2016-12-20 2017-05-31 维沃移动通信有限公司 一种拍摄方法及移动终端
CN109035185A (zh) * 2018-06-29 2018-12-18 努比亚技术有限公司 一种图像处理方法及终端
CN109167893A (zh) * 2018-10-23 2019-01-08 Oppo广东移动通信有限公司 拍摄图像的处理方法、装置、存储介质及移动终端

Also Published As

Publication number Publication date
CN113950705A (zh) 2022-01-18

Similar Documents

Publication Publication Date Title
US10867430B2 (en) Method and system of 3D reconstruction with volume-based filtering for image processing
US11838606B2 (en) Methods and systems for large-scale determination of RGBD camera poses
US10284789B2 (en) Dynamic generation of image of a scene based on removal of undesired object present in the scene
US9396399B1 (en) Unusual event detection in wide-angle video (based on moving object trajectories)
EP3296952B1 (fr) Procédé et dispositif pour corriger un objet virtuel dans une video
CN102148965B (zh) 多目标跟踪特写拍摄视频监控系统
CN109313799B (zh) 图像处理方法及设备
US20180182114A1 (en) Generation apparatus of virtual viewpoint image, generation method, and storage medium
WO2014023231A1 (fr) Système et procédé d'imagerie optique de très grande résolution et à large champ de vision
JP2018151689A (ja) 画像処理装置及びその制御方法、プログラム、記憶媒体
CN109981972B (zh) 一种机器人的目标跟踪方法、机器人及存储介质
CN111815517B (zh) 一种基于球机抓拍图片的自适应全景拼接方法
EP3629570A2 (fr) Appareil de capture d'images et procédé d'enregistrement d'images
CN111866523B (zh) 全景视频合成方法、装置、电子设备和计算机存储介质
US11812154B2 (en) Method, apparatus and system for video processing
CN112640419B (zh) 跟随方法、可移动平台、设备和存储介质
CN108010058A (zh) 一种对视频流中目标对象进行视觉跟踪的方法及系统
WO2022040988A1 (fr) Procédé et appareil de traitement d'images, et plateforme mobile
CN106469435B (zh) 图像处理方法、装置及设备
WO2021217403A1 (fr) Procédé et appareil de commande de plateforme mobile et dispositif et support de stockage
CN105467741A (zh) 一种全景拍照方法及终端
WO2020196520A1 (fr) Procédé, système, et supports lisibles par ordinateur pour estimer une couverture de détection d'objet
JP6483661B2 (ja) 撮像制御装置、撮像制御方法およびプログラム
KR101132976B1 (ko) 복수 개의 카메라를 구비한 모바일 기기, 이를 이용한 디스플레이 표시방법
JP2017021430A (ja) パノラマビデオデータの処理装置、処理方法及びプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20950660

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20950660

Country of ref document: EP

Kind code of ref document: A1