CN114693702A

CN114693702A - Image processing method, image processing device, electronic equipment and storage medium

Info

Publication number: CN114693702A
Application number: CN202210303750.0A
Authority: CN
Inventors: 王庆民; 赵雄
Original assignee: Xiaomi Automobile Technology Co Ltd
Current assignee: Xiaomi Automobile Technology Co Ltd
Priority date: 2022-03-24
Filing date: 2022-03-24
Publication date: 2022-07-01
Anticipated expiration: 2042-03-24
Also published as: CN114693702B

Abstract

The present disclosure relates to an image processing method, an apparatus, an electronic device, and a storage medium, the method including: acquiring a first shot image and a second shot image which are shot aiming at the same scene in sequence; processing the first shot image and the second shot image to obtain a first mask image comprising a first matte region and a second mask image comprising a second matte region; performing matting processing on the first shot image and the second shot image through the first mask image and the second mask image to obtain a plurality of sub-images; determining a target sub-image comprising a moving object in the scene from the plurality of sub-images; and determining the motion information of the moving object according to the shooting sequence of the shot images corresponding to the target sub-images and the position information of the cutout areas in the mask images for obtaining the target sub-images through cutout. The method and the device can accurately determine the motion condition of the moving object through the image and are simple to implement.

Description

Image processing method, image processing device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

Image processing techniques have been widely used in various fields, and most commonly, image processing techniques are used to detect a moving object and analyze the movement of the object by taking an image of the target object.

Disclosure of Invention

To overcome the problems in the related art, the present disclosure provides an image processing method, apparatus, electronic device, and storage medium.

According to a first aspect of embodiments of the present disclosure, there is provided an image processing method, including:

acquiring a first shot image and a second shot image which are shot aiming at the same scene in sequence;

processing the first shot image and the second shot image to obtain a first mask image comprising a first matte region and a second mask image comprising a second matte region;

for the first shot image and the second shot image, matting through the first mask image and the second mask image to obtain a plurality of sub-images;

determining, from the plurality of sub-images, a target sub-image comprising a moving object in the scene;

and determining the motion information of the motion object according to the shooting sequence of the shot images corresponding to the target sub-images and the position information of the cutout areas in the mask images for cutout obtaining the target sub-images.

Optionally, the processing the first captured image and the second captured image includes:

performing inter-frame difference processing on the first shot image and the second shot image to obtain a moving image of a moving object in the scene;

carrying out binarization processing on the moving image to obtain a binarization image comprising a first contour region and a second contour region;

and generating a first mask image comprising a first matte region and a second mask image comprising a second matte region according to the position information of the first contour region and the second contour region in the binarized image respectively, wherein the first contour region corresponds to the first matte region, and the second contour region corresponds to the second matte region.

Optionally, the determining, from the plurality of sub-images, a target sub-image including a moving object in the scene comprises:

and determining the sub-image meeting the similarity condition in the plurality of sub-images as the target sub-image.

Optionally, the determining, as the target sub-image, a sub-image that satisfies a similarity condition in the plurality of sub-images includes:

determining a similarity between each two of the plurality of sub-images;

and determining the two sub-images with the maximum similarity as the target sub-images.

Optionally, the obtaining a plurality of sub-images by performing matting processing on the first and second captured images through the first and second mask images includes:

aligning the first mask image and the first shot image, and scratching out a first sub-image from the first shot image according to the first scratching area;

aligning the first mask image and the second shot image, and scratching out a second sub-image from the second shot image according to the first scratching area;

aligning the second mask image and the first shot image, and scratching out a third sub-image from the first shot image according to the second scratching area;

and aligning the second mask image and the first shot image, and scratching out a fourth sub-image from the second shot image according to the second scratching area.

Optionally, the determining the motion information of the moving object according to the shooting order of the shot images corresponding to the target sub-images and the position information of the matting region in the mask image used for matting the target sub-images includes:

taking the position information of a cutout area corresponding to the target sub-image which is cutout from the first shot image as initial position information;

taking the position information of a cutout area corresponding to the target sub-image which is cutout from the second shot image as end point position information;

determining the motion direction and the motion track of the motion object according to the initial position information and the end position information;

and determining the motion direction and the motion trail as the motion information.

According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus including:

the shot image acquisition module is configured to acquire a first shot image and a second shot image which are shot aiming at the same scene in sequence;

a mask image acquisition module configured to process the first shot image and the second shot image to obtain a first mask image including a first matte region and a second mask image including a second matte region;

the matting module is configured to perform matting processing on the first shot image and the second shot image through the first mask image and the second mask image to obtain a plurality of sub-images;

a target sub-image determination module configured to determine a target sub-image comprising a moving object in the scene from the plurality of sub-images;

and the motion information determining module is configured to determine the motion information of the motion object according to the shooting sequence of the shot images corresponding to the target sub-images and the position information of the matting region in the mask image for matting the target sub-images.

According to a third aspect of an embodiment of the present disclosure, there is provided an electronic apparatus including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the image processing method provided by the first aspect of the present disclosure.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: acquiring a first shot image and a second shot image which are shot aiming at the same scene in sequence; processing the first shot image and the second shot image to obtain a first mask image comprising a first matte region and a second mask image comprising a second matte region; performing matting processing on the first shot image and the second shot image through the first mask image and the second mask image to obtain a plurality of sub-images; determining a target sub-image comprising a moving object in the scene from the plurality of sub-images; and determining the motion information of the moving object according to the shooting sequence of the shot images corresponding to the target sub-images and the position information of the cutout areas in the mask images for obtaining the target sub-images through cutout. Because the sub-image which is scratched through the first scratch area of the first mask image comprises a moving object, the position of the moving object which is in the first scratch area when the moving object is shot is shown, the position of the first scratch area when the sub-image which is scratched through the second scratch area of the first mask image comprises the moving object is shown, the position of the moving object which is in the first scratch area when the moving object is shot is shown, so that under the condition that a plurality of sub-images comprise target sub-images, the position of the first scratch area and the position of the second scratch area can be used as the positions of the moving object when the moving object is shot twice, and the shooting sequence of the shooting images corresponding to the target sub-images is combined, so that the information such as the moving direction and the moving track of the moving object can be determined, therefore, the moving information of the moving object can be accurately determined, the realization process is simple, and more power consumption of the execution equipment can not be spent.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a flow diagram illustrating an image processing method according to an exemplary embodiment.

Fig. 2 is a flowchart illustrating an image processing method according to another exemplary embodiment.

Fig. 3 is a schematic diagram of the first photographed image shown in the embodiment of fig. 2.

Fig. 4 is a schematic diagram of a second photographed image shown in the embodiment of fig. 2.

Fig. 5 is a schematic diagram of a moving image shown in the embodiment of fig. 2.

Fig. 6 is a schematic diagram of the binarized image shown in the embodiment of fig. 2.

Fig. 7 is a schematic diagram of the first mask image shown in the embodiment of fig. 2.

Fig. 8 is a schematic diagram of a second mask image shown in the embodiment of fig. 2.

Fig. 9 is a schematic diagram of the first sub-image shown in the embodiment of fig. 2.

Fig. 10 is a schematic diagram of the second sub-image shown in the embodiment of fig. 2.

Fig. 11 is a schematic diagram of a third sub-image shown in the embodiment of fig. 2.

Fig. 12 is a schematic diagram of a fourth sub-image shown in the embodiment of fig. 2.

Fig. 13 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

FIG. 14 is a block diagram illustrating an electronic device in accordance with an example embodiment.

FIG. 15 is a block diagram illustrating a server in accordance with an exemplary embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

It should be noted that all actions of acquiring signals, information or data in the present application are performed under the premise of complying with the corresponding data protection regulation policy of the country of the location and obtaining the authorization given by the owner of the corresponding device.

With the increasing Processing capacity of embedded Central Processing Units (CPUs) and Graphics Processing Units (GPUs), the digital image Processing capacity based on the processors is greatly increased, and various image functions such as beauty, character recognition, video editing, image editing, automatic driving and the like can be realized. The current commonly used technology is to determine the motion information of a moving object by processing the shot image of the moving object, and the technology can be applied to the fields of video monitoring, automatic driving, speed limit monitoring and the like, thereby reducing the labor cost to a great extent.

In the related art, the motion of a moving object is usually detected by using an object segmentation method and an optical flow method, but the above detection methods have certain defects.

For example, the object segmentation method is to acquire an image of an object in the real world, train a model, identify the object in the image through the model, and analyze the motion of the object. Because the trained model can only be used for the object as the training object and cannot cover all objects, when the object is detected, if the object is encountered except the object as the training object, the segmentation result will be inaccurate, so that the motion detection of the object is also inaccurate, and the process of model training is very complicated.

For another example, the optical flow method is a method of calculating motion information of an object between adjacent frames by finding a correspondence between a previous frame and a current frame using a change in a temporal region of pixels in an image sequence and a correlation between adjacent frames. However, this method has high requirements for device performance, high power consumption, and low accuracy.

In view of the foregoing problems, the present disclosure provides an image processing method, an image processing apparatus, an electronic device, and a storage medium, which are capable of accurately detecting a motion condition of a motion, and are simple to implement and low in power consumption.

Fig. 1 is a flowchart illustrating an image processing method according to an exemplary embodiment, which is used in a terminal, as shown in fig. 1, and includes the steps of:

in step S11, a first captured image and a second captured image captured in succession for the same scene are acquired.

Alternatively, the terminal may include, but is not limited to: personal computers, servers, smart phones, and the like have an image processing function.

In some embodiments, the terminal may receive the first photographed image and the second photographed image photographed and transmitted by the image photographing device. The image shooting device can keep the shooting position and the shooting angle unchanged during shooting, so that the image shooting device always shoots in the same scene, namely, the background of a shot image obtained by every shooting of the image shooting device is consistent except for a moving object.

Alternatively, the image capturing device may be disposed on the terminal, or may be disposed independently, which is not limited herein.

Optionally, the image capturing devices include, but are not limited to: high definition cameras, high speed cameras, and the like.

In step S12, the first captured image and the second captured image are processed to obtain a first mask image including a first matting region and a second mask image including a second matting region.

In some embodiments, the terminal may perform inter-frame difference processing on the first captured image and the second captured image to obtain object outlines of moving objects at different positions in the same moving image.

Then, the moving image including the object contour is subjected to binarization processing to filter out the part of the image except the object contour, and the position information of the object contour at different positions in the image is recorded to obtain a binarization image only containing the object contour at different positions.

And then according to the position information of the object outlines at different positions in the binary image, respectively placing the different object outlines into two images to respectively obtain a first mask image and a second mask image, wherein the first matting area in the first mask image corresponds to the object outline of one position information, and the second matting area in the second mask image corresponds to the object outline of the other position information.

It can be understood that, since the first mask image and the second mask image are obtained based on the position information of the object contour at different positions in the image, when the first mask image and the second mask image are overlapped, the binarized image including the object contour at different positions can be obtained.

Alternatively, the position information may be position coordinates, such as coordinates of the center of the contour of the object.

In step S13, for each of the first captured image and the second captured image, matting is performed using the first mask image and the second mask image to obtain a plurality of sub-images.

Illustratively, the terminal can perform matting processing on the first shot image through a first mask image to matte out a first sub-image corresponding to the first matting region from the first shot image; carrying out matting processing on the second shot image through the first mask image so as to matte out a second sub-image corresponding to the first matting region from the second shot image; carrying out matting processing on the first shot image through a second mask image so as to matte out a third sub-image corresponding to the second matting region from the first shot image; and carrying out matting processing on the second shot image through a second mask image so as to matte a fourth sub-image corresponding to the second matting region from the second shot image.

In step S14, a target sub-image including a moving object in the scene is determined from the plurality of sub-images.

Following the above example, the terminal may respectively perform motion object recognition on the first sub-image, the second sub-image, the third sub-image, and the fourth sub-image, and determine the sub-image in which the motion object is recognized from the plurality of sub-images as the target sub-image, for example, if the terminal recognizes that the first sub-image and the fourth sub-image include the motion object, the first sub-image and the fourth sub-image may be determined as the target sub-image.

In step S15, the motion information of the moving object is determined according to the shooting order of the shot images corresponding to the target sub-images and the position information of the matting regions in the mask images for matting the target sub-images.

Following the above example, since the first sub-image is cut out from the first photographed image, the second sub-image is cut out from the second photographed image, and the photographing time of the first photographed image is before the photographed image of the second photographed image, the position of the moving object corresponding to the first sub-image is between the positions of the moving objects corresponding to the first sub-image. Then, the motion information of the moving object can be determined according to the position of the moving object corresponding to the first sub-image (i.e., the position information of the first contour region) and the position of the moving object corresponding to the second sub-image (i.e., the position information of the second contour region).

An example is to determine the position of the first contour region toward the position of the first contour region as the moving direction of the moving object, for example, based on the position information of the first contour region and the position information of the first contour region.

For another example, a trajectory connecting the position of the first contour region and the position of the first contour region is determined as the motion trajectory of the moving object, for example, according to the position information of the first contour region and the position information of the first contour region.

In still another example, for example, the position information of the first contour region is determined as the moving position of the moving object corresponding to the capturing time of the first captured image. And determining the position information of the second contour area as the motion position of the motion object corresponding to the shooting time of the second shot image.

Optionally, the motion information may include other motion information such as distance, speed, and the like, in addition to the motion direction, the motion position, and the motion track, which is not limited herein.

As can be seen, in this embodiment, a first captured image and a second captured image obtained by sequentially capturing images of the same scene are obtained; processing the first shot image and the second shot image to obtain a first mask image comprising a first matte region and a second mask image comprising a second matte region; performing matting processing on the first shot image and the second shot image through the first mask image and the second mask image to obtain a plurality of sub-images; determining a target sub-image comprising a moving object in the scene from the plurality of sub-images; and determining the motion information of the moving object according to the shooting sequence of the shot images corresponding to the target sub-images and the position information of the cutout areas in the mask images for obtaining the target sub-images through cutout. Because the sub-image which is scratched through the first matting area of the first mask image comprises the moving object, the situation that the moving object is positioned at the position of the first matting area when being shot is indicated, and the situation that the sub-image which is scratched through the second matting area of the first mask image comprises the moving object is indicated that the moving object is positioned at the position of the first matting area when being shot, under the condition that a plurality of sub-images comprise target sub-images, the position of the first matting area and the position of the second matting area can be used as the positions of the moving object when being shot twice, and then the information such as the moving direction, the moving track and the like of the moving object can be determined by combining the shooting sequence of the shot images corresponding to the target sub-images, so that the moving information of the moving object can be accurately determined, more power consumption of the execution equipment can not be consumed, and high power consumption of the execution equipment is avoided, The requirement of high performance saves the tedious process of model training, and the realization is simple, and the application scope is wide.

Fig. 2 is a flowchart illustrating an image processing method according to another exemplary embodiment, which is used in a terminal, as illustrated in fig. 2, including the steps of:

in step S21, a first captured image and a second captured image captured in succession for the same scene are acquired.

Illustratively, for example, a first captured image acquired by the terminal is shown in fig. 3, and a second captured image acquired by the terminal is shown in fig. 4, where "duck" in fig. 3 and 4 is a moving object.

In step S22, an inter-frame difference process is performed on the first captured image and the second captured image to obtain a moving image of a moving object in the scene.

Following the above example, the terminal may perform the inter-frame difference processing on fig. 3 and 4, and since the inter-frame difference method is a method for obtaining the contour of the moving object by performing a difference operation on two adjacent frames in the video image sequence, after performing the inter-frame difference processing on fig. 3 and 4, the moving image as shown in fig. 5 may be obtained, where in fig. 5, the moving objects at different positions are included, that is, the position corresponding to the moving object in the first captured image and the position corresponding to the moving object in the second captured image.

In step S23, the moving image is binarized to obtain a binarized image including a first contour region and a second contour region.

Continuing with the above example, the terminal may perform binarization processing on fig. 5, for example, using, as the first pixel, a pixel of all pixels in fig. 5 whose gray value is greater than or equal to the gray value threshold, using, as the second pixel, a pixel of all pixels whose gray value is less than the gray value threshold, changing the gray value of the first pixel to 255, and changing the gray value of the second pixel to 0, so that the whole image exhibits an obvious black-and-white effect. Therefore, after the binarization processing of fig. 5, a binarized image as shown in fig. 6 can be obtained, and the data amount in fig. 6 is greatly reduced compared to fig. 5, so that the contour of the moving object can be highlighted.

In step S24, a first mask image including a first matte region and a second mask image including a second matte region are generated based on the position information of the first contour region and the second contour region in the binarized image, respectively, wherein the first contour region corresponds to the first matte region, and the second contour region corresponds to the second matte region.

Following the above example, it is apparent from fig. 6 that two contour regions are visible, and the terminal takes the contour region located on the left side of fig. 6 as the first contour region and records the position information of the first contour region in fig. 6. The terminal may also regard the outline area located at the right side of fig. 6 as a second outline area and record the position information of the second outline area in fig. 6. Then, according to the position information of the first contour region and the position information of the second contour region, the first contour region and the second contour region are respectively arranged in two mask images, for example, the first contour region is arranged in the first mask image, and the first contour region is used as a first matting region in the first mask image; the second outline region is set into the second mask image and serves as a second matting region in the second mask image. So that a first mask image as shown in fig. 7 and a second mask image as shown in fig. 8 can be obtained.

Wherein the first photographed image, the second photographed image, the first mask image, and the second mask image may be the same in size.

In step S25, for each of the first captured image and the second captured image, matting is performed using the first mask image and the second mask image to obtain a plurality of sub-images.

In some embodiments, a specific implementation of step S25 may include:

in step S251, the first mask image and the first captured image are aligned, and a first sub-image is extracted from the first captured image according to the first extraction region.

Following the above example, the terminal can align the rectangular edge of fig. 7 with the rectangular edge of fig. 3, overlay fig. 7 on fig. 3, and extract a first sub-image corresponding to the first cutout region from fig. 3, illustratively, the extracted first sub-image is shown in fig. 9.

In step S252, the first mask image and the second captured image are aligned, and a second sub-image is extracted from the second captured image according to the first extraction region.

Following the above example, the terminal can align the rectangular edge of fig. 7 with the rectangular edge of fig. 4, overlay fig. 4 with fig. 7, and extract a second sub-image corresponding to the first extracted region from fig. 4, illustratively, the extracted second sub-image is shown in fig. 10.

In step S253, the second mask image and the first captured image are aligned, and a third sub-image is extracted from the first captured image according to the second extraction region.

Following the above example, the terminal can align the rectangular edge of fig. 8 with the rectangular edge of fig. 3, overlay fig. 8 on fig. 3, and extract a third sub-image corresponding to the second matting area from fig. 3, illustratively, the extracted first sub-image is shown in fig. 11.

In step S254, the second mask image and the first captured image are aligned, and a fourth sub-image is extracted from the second captured image according to the second extraction region.

Following the above example, the terminal can align the rectangular edge of fig. 8 with the rectangular edge of fig. 4, overlay fig. 4 with fig. 8, and extract a fourth sub-image corresponding to the second matting area from fig. 4, illustratively, the extracted fourth sub-image is shown in fig. 12.

In step S26, a target sub-image including a moving object in the scene is determined from the plurality of sub-images.

In some embodiments, specific embodiments of step S26 may include:

in step S261, a sub-image satisfying the similarity condition among the plurality of sub-images is determined as the target sub-image.

As one way, a specific implementation of step S261 may include:

determining the similarity between every two sub-images in the plurality of sub-images; and determining the two sub-images with the maximum similarity as the target sub-images.

Following the above example, the terminal may calculate the similarity between each two images in fig. 9, 10, 11, and 12, and then determine the two sub-images with the greatest similarity as the target sub-images. For example, the similarity between fig. 9 and 12 is calculated to be the largest, and since the similarity between two sub-images including a moving object may be greater than the similarity between two sub-images not including a moving object, fig. 9 and 12 may be determined as the target sub-image.

Alternatively, the terminal may determine the similarity between fig. 9 and fig. 12 to obtain the similarity 1, determine the similarity between fig. 10 and fig. 11 to obtain the similarity 2, and use the sub-image corresponding to the maximum similarity between the similarity 1 and the similarity 2 as the target sub-image, for example, if the similarity 1 is greater than the similarity 2, then fig. 9 and fig. 12 may be used as the target sub-image, as can be seen from fig. 9 and fig. 12, where the moving object (duck) exists.

As another mode, the terminal may acquire a standard image including the captured image in a case where the moving object in the captured image is recognized, then compare the similarity of the plurality of sub-images with the standard image, respectively, and use the first two sub-images with the greatest similarity with the standard image among the plurality of sub-images as the target sub-images.

In view of the fact that two sub-images including a moving object and having a greater similarity between the sub-images including the moving object are inevitably present in the four sub-images obtained by matting the first shot image and the second shot image with respect to the first shot image and the second shot image, in the present embodiment, the similarity comparison is performed on the plurality of sub-images, and the sub-image having the greater similarity is determined as the target sub-image, so that the target sub-image can be accurately and simply determined.

In step S27, the motion information of the moving object is determined according to the shooting order of the shot images corresponding to the target sub-images and the position information of the matting regions in the mask images for matting the target sub-images.

In some embodiments, specific embodiments of step S27 may include:

step S271, using the position information of the cutout area corresponding to the target sub-image cutout from the first captured image as the start position information.

Following the above example, since the target sub-image (fig. 9) is extracted from the first extracted region of the first captured image (fig. 3) and the capturing time of the first captured image is before the second captured image, the position information of the first extracted region can be used as the start position information of the moving object.

In step S272, the position information of the cutout region corresponding to the target sub-image extracted from the second captured image is used as the end point position information.

Following the above example, since the object sub-image (fig. 12) is scratched out of the second matte region of the first shot image (fig. 4) and the shooting time of the second shot image is after the first shot image, it can be determined that the moving object is moved from the first matte region to the second matte region, and then the position information of the second matte region can be taken as the end point position information of the moving object.

Step S273, determining the motion direction and motion trajectory of the moving object according to the start position information and the end position information.

In step S274, the motion direction and the motion trajectory are determined as the motion information.

Alternatively, the length of the shooting interval between the first shot image and the second shot image is set according to the speed of movement of the moving object in the scene.

In some embodiments, a duration of a shooting interval between the first shot image and the second shot image may be inversely related to a moving speed of a moving object in the scene.

Considering that the moving speed of the moving object is slow, and the shooting interval duration is short, the moving object overlapping condition may occur in the moving image obtained by performing inter-frame difference on the first shot image and the second shot image; if the moving speed of the moving object is too fast, the shooting interval is too long, the distances between the moving objects at different positions in the moving image will be too different, and the motion detection will be inaccurate. In this embodiment, the shooting interval duration between the first shot image and the second shot image is set according to the motion speed of the moving object in the scene, so that the shooting interval duration of the images can be ensured to be matched with the motion speed of the moving object, and therefore, the terminal can determine the motion information of the moving object more accurately according to the obtained first shot image and the obtained second shot image.

It can be seen that, in this embodiment, the mask image is extracted from the captured image, and when the image including the moving object can be extracted from the captured image through the mask image, the position information of the extraction area of the mask image is used as the position information of the moving object, and then the movement information of the moving object is obtained according to the position information of the moving object, so that the movement information of the moving object can be simply, effectively and accurately obtained.

Fig. 13 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. Referring to fig. 13, the apparatus 30 includes a photographed image acquiring module 31, a mask image acquiring module 32, a matting module 33, an object sub-image determining module 34, and a motion information determining module 35. Wherein:

the captured image acquiring module 31 is configured to acquire a first captured image and a second captured image captured successively with respect to the same scene.

A mask image obtaining module 32 configured to process the first captured image and the second captured image to obtain a first mask image including a first matting region and a second mask image including a second matting region.

A matting module 33 configured to perform matting on the first captured image and the second captured image to obtain a plurality of sub-images through the first mask image and the second mask image;

a target sub-image determining module 34 configured to determine a target sub-image including a moving object in the scene from the plurality of sub-images.

And the motion information determining module 35 is configured to determine the motion information of the motion object according to the shooting order of the shot images corresponding to the target sub-images and the position information of the matting regions in the mask images for matting the target sub-images.

In some embodiments, the mask image acquisition module 32 includes:

and a moving image determination sub-module configured to perform an inter-frame difference process on the first captured image and the second captured image to obtain a moving image of a moving object in the scene.

And the binarized image determining submodule is configured to perform binarization processing on the moving image to obtain a binarized image comprising a first contour region and a second contour region.

And a mask image generation submodule configured to generate a first mask image including a first matting area and a second mask image including a second matting area according to position information of the first contour area and the second contour area in the binarized image, respectively, wherein the first contour area corresponds to the first matting area, and the second contour area corresponds to the second matting area.

In some embodiments, the target sub-image determination module 34 is specifically configured to:

and determining the sub-image satisfying the similarity condition in the plurality of sub-images as the target sub-image.

In some embodiments, the target sub-image determination module 34 is specifically configured to: determining the similarity between every two sub-images in the plurality of sub-images; and determining the two sub-images with the maximum similarity as the target sub-images.

In some embodiments, the matting module 33 includes:

and the first matting submodule is configured to align the first mask image and the first shot image and matte a first sub-image from the first shot image according to the first matting area.

And the second matting submodule is configured to align the first mask image and the second shot image and matte a second sub-image from the second shot image according to the first matting area.

And the third matting submodule is configured to align the second mask image and the first shot image and matting a third sub-image from the first shot image according to the second matting area.

And the fourth matting submodule is configured to align the second mask image and the first shot image and matting a fourth sub-image from the second shot image according to the second matting region.

In some embodiments, the motion information determining module 35 is specifically configured to: taking the position information of a cutout area corresponding to the target sub-image which is cutout from the first shot image as initial position information; taking the position information of the cutout area corresponding to the target sub-image which is cutout from the second shot image as the end point position information; determining the motion direction and the motion track of the motion object according to the initial position information and the end position information; and determining the motion direction and the motion trail as the motion information.

In some embodiments, the photographing interval duration between the first photographed image and the second photographed image is set according to a moving speed of a moving object in the scene.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

The present disclosure also provides a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the image processing method provided by the present disclosure.

Fig. 14 is a block diagram illustrating an electronic device 800 for an image processing method according to an example embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a tablet device, and the like.

Referring to fig. 14, electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power components 806 provide power to the various components of the electronic device 800. Power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 804 comprising instructions, executable by the processor 820 of the electronic device 800 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the image processing method described above when executed by the programmable apparatus.

Fig. 15 is a block diagram illustrating a server 1900 for image processing according to an example embodiment. For example, server 1900 may be provided as a server. Referring to FIG. 15, the server 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, executable by the processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The server 1900 may further include a power component 1926 configured to perform power management of the server 1900, a wired or wireless network interface 1950 configured to connect the server 1900 to a network, and an input/output (I/O) interface 1958. Server 1900 may operate based on data stored in memory 1932Operating systems, e.g. Windows Server^TM，Mac OS X^TM，Unix^TM，Linux^TM，FreeBSD^TMOr the like.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, comprising:

2. The image processing method according to claim 1, wherein the processing the first captured image and the second captured image includes:

performing interframe difference processing on the first shot image and the second shot image to obtain a moving image of a moving object in the scene;

3. The method of claim 1, wherein determining a target sub-image from the plurality of sub-images that includes a moving object in the scene comprises:

4. The method of claim 3, wherein determining the sub-image of the plurality of sub-images that satisfies the similarity condition as the target sub-image comprises:

determining a similarity between each two of the plurality of sub-images;

5. The method according to claim 1, wherein the matting the first and second captured images by the first and second mask images to obtain a plurality of sub-images comprises:

aligning the first mask image and the first shot image, and matting out a first sub-image from the first shot image according to the first matting area;

6. The image processing method according to claim 1, wherein the determining the motion information of the moving object according to the shooting order of the shot images corresponding to the target sub-images and the position information of the matting region in the mask image for matting the target sub-images comprises

7. The image processing method according to any one of claims 1 to 6, characterized in that a photographing interval duration between the first photographed image and the second photographed image is set according to a moving speed of a moving object in the scene.

8. An image processing apparatus characterized by comprising:

and the motion information determining module is configured to determine the motion information of the motion object according to the shooting sequence of the shot images corresponding to the target sub-images and the position information of the matting regions in the mask images for matting the target sub-images.

9. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

10. A computer-readable storage medium, on which computer program instructions are stored, which program instructions, when executed by a processor, carry out the steps of the method according to any one of claims 1 to 7.