CN113012272A

CN113012272A - Image processing method and device, electronic equipment and storage medium

Info

Publication number: CN113012272A
Application number: CN202110348514.6A
Authority: CN
Inventors: 冯忠伟; 冯巍; 宇哲伦; 姚文正; 贾霖
Original assignee: Beijing QIYI Century Science and Technology Co Ltd
Current assignee: Beijing QIYI Century Science and Technology Co Ltd
Priority date: 2021-03-31
Filing date: 2021-03-31
Publication date: 2021-06-22

Abstract

The embodiment of the invention provides an image processing method, an image processing device, electronic equipment and a storage medium, and relates to the technical field of image processing, wherein the method comprises the following steps: acquiring an image frame to be processed; carrying out object recognition on the image frame to be processed, and determining an object area where an object in the image frame to be processed is located; determining at least one region to be adjusted in the image frame to be processed; and determining a target area, wherein the target area belongs to the area to be adjusted and does not belong to the object area, and adjusting the pixel value of a pixel point in the target area to obtain the image frame with the adjusted pixel value. By applying the scheme provided by the embodiment of the invention to process the image, the convenience of obtaining the image with the 3D effect which can be observed by the user can be improved.

Description

Image processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

With the rapid development of video technology, in order to provide better viewing and listening experience for users and increase the viewing interest of the users, each video service provider expects to display an image and then the users can view the image with a 3D effect.

In the prior art, a mode of adding a lenticular lens in front of a display screen of an electronic device is generally adopted to process an image to be displayed, so that a user can view the image with a 3D effect. However, most electronic devices for users are not equipped with a lenticular lens, and therefore, processing an image in the above manner is not convenient enough to allow the user to view the image with a 3D effect.

Disclosure of Invention

An object of embodiments of the present invention is to provide an image processing method, an image processing apparatus, an electronic device, and a storage medium, so as to improve convenience in obtaining an image that enables a user to view a 3D effect. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides an image processing method, where the method includes:

acquiring an image frame to be processed;

carrying out object recognition on the image frame to be processed, and determining an object area where an object in the image frame to be processed is located;

determining at least one region to be adjusted in the image frame to be processed;

and determining a target area, wherein the target area belongs to the area to be adjusted and does not belong to the object area, and adjusting the pixel value of a pixel point in the target area to obtain the image frame with the adjusted pixel value.

In an embodiment of the present invention, the determining at least one region to be adjusted in the image frame to be processed includes:

determining at least one region to be adjusted in the edge region of the image frame to be processed;

and/or

And determining at least one area to be adjusted along the width direction and/or the height direction of the image frame to be processed.

In another embodiment of the present invention, the determining at least one region to be adjusted in the edge region of the image frame to be processed includes:

obtaining a region selection template, wherein the region selection template comprises: a shape parameter;

and determining at least one region to be adjusted in the edge region of the image frame to be processed based on the shape parameter.

In an embodiment of the present invention, the adjusting the pixel value of the pixel point in the target region to obtain the image frame after the pixel value adjustment includes:

adjusting the pixel value of a pixel point in the target area according to at least one of the following modes:

adjusting the pixel value of a pixel point in the target area to be a first preset pixel value;

reducing the value of the transparency component in the pixel value of the pixel point in the target area;

performing blurring processing on the target area;

adjusting the pixel value of the region edge pixel point to a second preset pixel value, wherein the region edge pixel point is as follows: and the pixel points in the image frame are positioned at the boundary of the target area and the non-target area.

In an embodiment of the present invention, the performing object identification on the image frame to be processed and determining an object region where an object is located in the image frame to be processed includes:

carrying out object identification on the image frame to be processed, and determining pixel points belonging to an object in the image frame to be processed;

generating a binary image which has the same size with the image frame to be processed, the pixel value of an object pixel point is a third preset pixel value, and the pixel value of a non-object pixel point is a fourth preset pixel value, wherein the object pixel point is as follows: the positions of the pixel points are the same as those of the pixel points belonging to the object in the image frame to be processed, and the non-object pixel points are as follows: pixels other than the object pixel;

the determining a target area and adjusting the pixel values of the pixel points in the target area to obtain the image frame with the adjusted pixel values includes:

adjusting pixel values of pixel points in the region to be adjusted in the image frame to be processed to obtain a temporary image frame;

and obtaining the image frame of which the pixel values of the pixels in the object region are the same as those of the pixels in the object region in the image frame to be processed before adjustment and the pixel values of the pixels in the non-object region are the same as those of the pixels in the non-object region in the temporary image frame according to the binary image, the image frame to be processed before adjustment of the pixel values and the temporary image frame.

In another embodiment of the present invention, in a case that the third preset pixel value is 1 and the fourth preset pixel value is 0, the obtaining, according to the binary image, the image frame to be processed before the pixel value adjustment and the temporary image frame, the image frame in which the pixel value of the pixel point in the object region is the same as the pixel value of the pixel point in the object region in the image frame to be processed before the adjustment and the pixel value of the pixel point in the non-object region is the same as the pixel value of the pixel point in the non-object region in the temporary image frame includes:

according to the pixel points, carrying out image multiplication calculation on the binary image and the image frame to be processed before the pixel value adjustment to obtain a first intermediate image;

carrying out pixel value negation operation on the binary image to obtain a second intermediate image;

according to pixel points, carrying out image multiplication calculation on the second intermediate image and the temporary image frame to obtain a third intermediate image;

and according to the pixel points, carrying out image addition calculation on the first intermediate image and the third intermediate image to obtain an image frame.

carrying out multilayer convolution transformation on the image frame to be processed to obtain a convolved image and the image characteristics of the image frame to be processed, which are extracted by each layer of convolution transformation;

determining a first object pixel point belonging to an object in the convolved image;

performing multilayer deconvolution transformation on the convolved image based on the image characteristics to obtain a deconvoluted image;

selecting second object pixel points corresponding to the first object pixel points from the deconvoluted image;

determining a region including a third object pixel point in the image frame to be processed as an object region where an object is located, wherein the third object pixel point is: and the pixel point with the same position as the second object pixel position in the image to be processed.

In a second aspect, an embodiment of the present application provides an image processing apparatus, including:

the image frame acquisition module is used for acquiring an image frame to be processed;

the object area determining module is used for carrying out object identification on the image frame to be processed and determining an object area where an object in the image frame to be processed is located;

a to-be-adjusted region determining module, configured to determine at least one to-be-adjusted region in the to-be-processed image frame;

and the pixel adjusting module is used for determining a target area, wherein the target area belongs to the area to be adjusted and does not belong to the object area, and adjusting the pixel values of the pixels in the target area to obtain the image frame with the adjusted pixel values.

In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;

a memory for storing a computer program;

a processor for implementing the method of any of the first aspects when executing a program stored in the memory.

In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the method of any one of the first aspect.

As can be seen from the above, when an image is processed by applying the scheme provided by the embodiment of the present invention, after an image frame to be processed is obtained, object identification is performed on the image frame to be processed, an object region where an object in the image to be processed is located is determined, at least one region to be adjusted in the image frame to be processed is determined, then, based on the object region and the region to be adjusted, a target region which belongs to the region to be adjusted but does not belong to the object region is obtained, and pixel values of pixel points in the target region are adjusted, so that the image frame to be processed is processed. The image frame to be processed is processed as above, so that not only the pixel values of the pixel points in the object region where the object is located in the image frame to be processed are reserved, but also the pixel values of the pixel points in the target region in the image frame to be processed are adjusted, and thus the adjusted image frame to be processed has a visual extension effect, and a user can view an image with a 3D effect. As can be seen from the above process, when the scheme provided by the embodiment of the present invention is applied to process an image, there is no need to add an auxiliary device such as a lenticular lens in front of a display screen of an electronic device, so that, compared with the prior art, the scheme provided by the embodiment of the present invention can improve the convenience of obtaining an image that enables a user to view the image with a 3D effect.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

Fig. 1 is a schematic flowchart of a first image processing method according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a second image processing method according to an embodiment of the present invention;

FIG. 3a is a diagram illustrating an image frame after a first pixel value adjustment according to an embodiment of the present invention;

FIG. 3b is a diagram illustrating an image frame after a second pixel value adjustment according to an embodiment of the present invention;

FIG. 3c is a diagram illustrating an image frame after a third pixel value adjustment according to an embodiment of the present invention;

FIG. 3d is a diagram illustrating a fourth image frame after adjusting pixel values according to an embodiment of the present invention;

FIG. 4 is a flowchart illustrating a third image processing method according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of a convolutional neural network according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a first image processing apparatus according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a second exemplary image processing apparatus according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.

The embodiment of the invention provides an image processing method and device for solving the technical problem that the convenience for a user to view an image with a 3D effect is poor by applying the prior art.

In an embodiment of the present invention, there is provided an image processing method, including:

acquiring an image frame to be processed;

carrying out object identification on the image frame to be processed, and determining an object area where an object in the image frame to be processed is located;

determining at least one region to be adjusted in an image frame to be processed;

As can be seen from the above, when the scheme provided by the embodiment of the present invention is applied to process an image, not only the pixel values of the pixels in the object region where the object is located in the image frame to be processed are retained, but also the pixel values of the pixels in the target region in the image frame to be processed are adjusted, so that the adjusted image frame to be processed has a visual extension effect, and thus, an image with a 3D effect can be viewed by a user. As can be seen from the above process, when the scheme provided by the embodiment of the present invention is applied to process an image, there is no need to add an auxiliary device such as a lenticular lens in front of a display screen of an electronic device, so that, compared with the prior art, the scheme provided by the embodiment of the present invention can improve the convenience of obtaining an image that enables a user to view the image with a 3D effect.

The following first describes the implementation subject of the solution provided in the embodiment of the present invention.

In one case, the execution subject of the embodiment of the present invention may be a server, and in this case, the server applies the scheme provided by the embodiment of the present invention to process the image, and then may transmit the processed image to the client, so that the client may perform operations such as displaying the received image.

In another case, the execution subject of the embodiment of the present invention may be a client, and in this case, the client applies the scheme provided by the embodiment of the present invention to process the image, and then directly performs operations such as displaying the processed image.

The following describes the image processing method, apparatus, electronic device and storage medium provided by the embodiments of the present invention in detail through specific embodiments.

Referring to fig. 1, a flow diagram of a first image processing method is provided, the method comprising the following steps S101-S104.

Step S101: and acquiring an image frame to be processed.

Specifically, the image frame to be processed may be an independent image, and in this case, after the image frame to be processed is processed by using the scheme provided by the embodiment of the present invention, an independent image capable of enabling a user to view an image with a 3D effect is obtained.

The image frame to be processed may also be an image frame in a video, and in this case, the image frame to be processed has temporal relevance to other image frames in the video, and after the scheme provided by the embodiment of the present invention is used to process a plurality of image frames in the video, a plurality of images capable of enabling a user to view the images with a 3D effect are obtained. Since the obtained images having a 3D effect have temporal correlation, it can be considered that a video having a 3D effect is obtained.

Step S102: and carrying out object identification on the image frame to be processed, and determining an object area where an object in the image frame to be processed is located.

Specifically, when the object identification is performed on the image frame to be processed, so as to determine the object region where the object is located in the image frame to be processed, various manners may be adopted. For example, the object recognition may be implemented by a matting method, an object segmentation method, a saliency detection method, a depth estimation method, or the like. Besides, the neural network model for object recognition can be trained in advance to realize object recognition. The neural network model may be a convolutional neural network model. The structure of the convolutional neural network model for object recognition is explained in the following embodiments, and will not be described in detail here.

In addition, the object region may be understood as: the region where the object is located in the image to be processed. The object region may have various presentation forms.

In one case, the object region may be represented by a mask image including two types of pixels, where one type of pixels is pixels belonging to the object and the other type of pixels is pixels not belonging to the object. Thus, the object area in the image to be processed can be determined according to the type of the pixel points in the mask image.

Alternatively, the object region may be represented by a coordinate sequence. And the coordinate sequence comprises coordinate information of pixel points belonging to the object in the image frame to be processed. Thus, the area where the object is located in the image frame to be processed can be directly determined based on the coordinates in the coordinate sequence.

In another case, the object region may be represented by coordinate information of edge pixel points belonging to the object in the image frame to be processed. In this case, the region surrounded by the edge pixel points in the image frame to be processed may be considered as the object region.

For example, the object may be a human, an animal, a vehicle, or the like.

Step S103: at least one region to be adjusted in the image frame to be processed is determined.

As will be understood by those skilled in the art, there is often spatial correlation between pixels in an image, so that for a local area in the image, the difference between pixel values of each pixel is generally small and changes slowly, thereby easily providing a flattened visual effect to a user. In this step, at least one region to be adjusted in the image frame to be processed is determined, that is, it means that the pixel values of the pixels in the region to be adjusted can be adjusted, so that, when viewed from the local region, the pixel values of the pixels in the region to be adjusted and the pixel values of the pixels in the adjacent region generate a larger difference, so that a user can feel the change of the pixel values in the local region in the image when viewing the image, thereby the spatial sense of the user for the image content is enhanced, the planar sense is weakened, and the visual extension sense of the user when viewing the image to be processed is enhanced.

Step S104: and determining a target area, wherein the target area belongs to the area to be adjusted and does not belong to the object area, and adjusting the pixel value of a pixel point in the target area to obtain the image frame with the adjusted pixel value.

Specifically, since the target area belongs to the area to be adjusted and does not belong to the object area, when the target area is determined, the target area may be determined in a direct determination manner or in an indirect determination manner.

If the target area is determined in a direct determination mode, the target area can be determined in a mode of detecting pixels in the area to be adjusted, if the pixels in the area to be adjusted do not belong to the pixels in the target area, the pixels belong to the pixels in the target area, if the pixels in the area to be adjusted belong to the pixels in the target area, the pixels do not belong to the pixels in the target area, and the target area is finally determined according to the detected pixels which belong to the target area.

When the target area is determined in an indirect determination mode, the position of the target area in the image frame to be processed can be determined in an undefined mode. The specific implementation manner can be seen in the following embodiments, and detailed description is omitted here.

When the pixel value is adjusted, the determined target area belongs to the area to be adjusted and does not belong to the object area, namely, on the premise of keeping the pixel value of the pixel point in the object area, the pixel value of the pixel point in the area to be adjusted is adjusted, so that the object presented to the user can be more prominent after the pixel value of the pixel point in the target area is changed, thereby further enhancing the three-dimensional effect of the object, namely, further waiting for the visual extension sense of the user when viewing the image to be processed to be enhanced.

Specifically, the image frame after the pixel value adjustment may be obtained by adjusting the pixel value of the pixel point in the target region in at least one of the following manners.

The first method is as follows: and adjusting the pixel value of the pixel point in the target area to be a first preset pixel value.

For example, the first preset pixel value may be a pixel value indicating black, such as 0, or a pixel value indicating white, such as 255.

Of course, the pixel values of the pixel points in the target region may also be adjusted to other pixel values, which is not limited in the embodiment of the present invention.

The second method comprises the following steps: and reducing the value of the transparency component in the pixel value of the pixel point in the target area.

Specifically, the transparency component in the pixel value of the pixel point in the target area can be reduced to the same transparency. For example, the same transparency as above may be 0, in which case it may be considered to be completely opaque; the same transparency as above may also be 0.5, in which case it may be considered translucent.

The third method comprises the following steps: and performing blurring processing on the target area.

The blurring process is a process of blurring the target region.

Specifically, each pixel in the target area may be subjected to blurring processing by using the pixel as a unit. When each pixel point is subjected to blurring processing, blurring processing can be realized based on the pixel point and pixel values of pixel points around the pixel point. For example, the weighting coefficient of each surrounding pixel point may be set according to the distance between each surrounding pixel point and the pixel point, the weighting coefficient of the pixel point may be set, and then, based on each weighting coefficient, the pixel value of the pixel point and the pixel values of the surrounding pixel points may be weighted and calculated to obtain the pixel value after the blurring processing of the pixel point.

The above-mentioned surrounding pixels may be around the pixel, the first number of pixels may be around the pixel, the second number of pixels may be around the pixel, or the adjacent pixels around the pixel in 8 directions may be around the pixel.

For example, the first and second numbers may be 2, 3, etc.

The method is as follows: adjusting the pixel value of the region edge pixel point to a second preset pixel value, wherein the region edge pixel point is as follows: and pixel points in the image frame, which are located at the junction of the target area and the non-target area.

The target area may be an edge area in the image frame to be processed, that is, one boundary of the target area belongs to an image edge of the image frame to be processed, and the target area may also be a non-edge area in the image frame to be processed, that is, the boundary of the target area does not belong to the image edge of the image frame to be processed at all.

And when the target area is an edge area in the image frame to be processed, the area edge pixel points represent pixel points which are not located in the boundary of the image edge of the image frame to be processed and are located at the boundary of the non-target area in the target area.

And when the target area is a non-edge area in the image frame to be processed, the area edge pixel points represent pixel points of the whole boundary of the target area and the boundary of the non-target area.

The second predetermined pixel value may be determined according to pixel values of pixel points in the target region and/or the non-target region. Based on this, in one implementation, the second predetermined pixel value may be a pixel value having a difference from a pixel value in the target region larger than a first predetermined difference, and/or the second predetermined pixel value may be a pixel value having a difference from a pixel value of a pixel point in the non-target region larger than a second predetermined difference.

For example, if the pixel value of the pixel point in the target region is a pixel value corresponding to black, the second preset pixel value may be a pixel value corresponding to white; for another example, if the pixel value of the pixel point in the target region is black and the pixel value of the pixel point in the non-target region is white, the second predetermined pixel value may be a pixel value corresponding to gray.

The pixel values of the pixel points at the edge of the region are adjusted according to the rule, so that the pixel values of the pixel points in the target region, the pixel values of the pixel points in the non-target region and the pixel values of the pixel points at the edge of the region have larger difference, and the spatial sense of the user for the image content is further enhanced.

In an embodiment of the present invention, the step S103 of determining at least one region to be adjusted in the image frame to be processed includes the following step S103A.

Step S103A: at least one region to be adjusted is determined in an edge region of an image frame to be processed.

Specifically, the edge region of the image frame to be processed may be a region away from the boundary of the image frame to be processed by a preset distance, and the boundary of the image frame to be processed may be four sides, i.e., the uppermost pixel row, the lowermost pixel row, the leftmost pixel column, and the rightmost pixel column of the image frame to be processed.

In this case, the determined region to be adjusted may be the entire edge region, or may be a partial region in the edge region, and if the determined region to be adjusted is the entire edge region, the region to be adjusted is a regular region; if the determined region to be adjusted is a partial region in the edge region, the region to be adjusted may be a regular region or an irregular region.

The preset distance may be 15 pixels, 20 pixels, and so on.

In another embodiment of the present invention, the step S103A determining at least one region to be adjusted in the edge region of the image frame to be processed includes the following steps S103a1-S103a 2.

Step S103a 1: a region selection template is obtained.

Wherein the region selection template comprises: a shape parameter.

The area selection template may be a frame with a specific shape edge, or may be an image for determining the designated area, where the shape parameter may be a specific value of the width and height of the frame or the designated area in the area selection template, or may be an aspect ratio of the frame or the designated area.

The special-shaped side can be regular, such as rectangular, or irregular, such as sawtooth, square wave.

In addition, the size of the region selection template may be the same as the size of the image frame to be processed, or may be different from the size of the image frame to be processed.

For example, the shape parameter of the region selection template may be a specific value that specifies the width and height of the region when the size of the region selection template is the same as the size of the image frame to be processed, and the shape parameter of the region selection template may be an aspect ratio of the specified region when the size of the region selection template is different from the size of the image frame to be processed.

Step S103a 2: and determining at least one region to be adjusted in the edge region of the image frame to be processed based on the shape parameter.

Since the shape parameter records the specific values of the width and the height or the aspect ratio, the region to be adjusted can be determined according to the specific values of the width and the height or the aspect ratio in the edge region of the image frame to be processed. And, if the shape parameter records a plurality of specific values of width and height, or aspect ratio, a plurality of regions to be adjusted may be determined in the edge region of the image frame to be processed.

As can be seen from the above, in the solution provided by this embodiment, at least one region to be adjusted may be determined in the edge region of the image frame to be processed according to the shape parameter of the region selection template. The shape parameters of the area selection template comprise information representing the area selection template, and the area to be adjusted, which is determined in the edge area of the image frame to be processed according to the shape parameters, is also determined according to the information in the area selection template, namely, the area selection template is a template with a specific shape, the shape parameters are used for recording the shape characteristics of the area selection template, and the area to be adjusted, which is determined based on the shape parameters, also has the shape characteristics of the area selection template. Therefore, different regions to be adjusted can be determined in the edge region of the image frame to be processed by selecting different region selection templates.

In another embodiment of the present invention, the step S103 of determining at least one region to be adjusted in the image frame to be processed includes the following step S103B.

Step S103B: at least one region to be adjusted is determined in the width direction and/or the height direction of the image frame to be processed.

In this case, the determined region to be adjusted may be in an edge region of the image frame to be processed, or may be in a non-edge region of the image frame to be processed.

Taking the determination of at least one to-be-adjusted region along the width direction of the to-be-processed image frame as an example, if the determined to-be-adjusted region is an edge region of the to-be-processed image frame, the to-be-adjusted region includes the uppermost pixel row or the lowermost pixel row of the to-be-processed image frame, and the width is a preset width.

If the determined area to be adjusted is a non-edge area of the image frame to be processed, the area to be adjusted does not include the uppermost pixel row and the lowermost pixel row of the image frame to be processed, and the width of the area to be adjusted is a preset width.

The region to be adjusted may be a single region having a continuous row, or may be a discrete region having a plurality of continuous rows.

The preset width may be 5 pixels, 8 pixels, and so on.

Similarly, when at least one region to be adjusted is determined along the height direction of the image frame to be processed, the region to be adjusted may be in an edge region of the image frame to be processed, may be in a non-edge region of the image frame to be processed, may be a single region with continuous lines, or may be a discrete region with multiple continuous lines.

In one case, the region to be adjusted is determined in the width direction or the height direction of the image frame to be processed, and the region to be adjusted is a regular region.

In another case, the region to be adjusted is determined along the width direction and the height direction of the image frame to be processed, and the region to be adjusted is an irregular region.

In one implementation, the region to be adjusted may be determined according to the methods shown in step S103A and step S103B at the same time.

As can be seen from the above, in the solution provided by this embodiment, at least one region to be adjusted may be determined in the edge region of the image frame to be processed, or at least one region to be adjusted may be determined along the width direction and/or the height direction of the image frame to be processed, or a plurality of regions to be adjusted may be determined in combination with the edge region of the image frame to be processed and the width direction and/or the height direction of the image frame to be processed, so that a plurality of methods for determining the region to be adjusted are provided, and the diversity of image processing is improved.

In an embodiment of the present invention, referring to fig. 2, a flowchart of a second image processing method is provided, and compared with the foregoing embodiment shown in fig. 1, in this embodiment, the step S102 of performing object recognition on the image frame to be processed to determine the object region where the object is located in the image frame to be processed includes the following steps S102a1-S102B 1:

step S102a 1: and carrying out object identification on the image frame to be processed, and determining pixel points belonging to the object in the image frame to be processed.

Step S102B 1: generating a binary image which has the same size with the image frame to be processed, the pixel value of the object pixel point is a third preset pixel value, and the pixel value of the non-object pixel point is a fourth preset pixel value, wherein the object pixel point is as follows: the positions of the pixel points are the same as those of the pixel points belonging to the object in the image frame to be processed, and the non-object pixel points are as follows: pixels other than the object pixel.

Since the size of the binary image is the same as that of the image frame to be processed, and the positions of the object pixel points are the same as those of the pixel points belonging to the object determined in step S102a1, the object pixel points in the binary image can represent the pixel points of the object in the image frame to be processed.

Specifically, the third predetermined pixel value is different from the fourth predetermined pixel value. When the pixel value of one pixel is represented by 2 bits, the third preset pixel value may be 0 or 1, and correspondingly, the fourth preset pixel value may be 1 or 0. Similarly, when the pixel value of one pixel is represented by 8 bits, the pixel value of the third preset pixel may be 0 or 255, and correspondingly, the fourth preset pixel may be 255 or 0.

In this embodiment, the step S104 of determining the target area and adjusting the pixel values of the pixels in the target area to obtain the image frame with the adjusted pixel values includes the following steps S104A-S104B.

Step S104A: and adjusting the pixel value of a pixel point in the region to be adjusted in the image frame to be processed to obtain a temporary image frame.

Specifically, when the pixel value of the pixel point in the region to be adjusted is adjusted, at least one of the first to fourth manners may be adopted, and details are not repeated here.

Based on the above situation, in an embodiment of the present invention, the temporary image frame may be obtained by adjusting the transparency of the pixel point in the to-be-adjusted region in the to-be-processed image frame and adjusting the pixel value of the pixel point in the to-be-adjusted region in the to-be-processed image frame.

For example, the transparency of the pixel point in the region to be adjusted may be adjusted to 0.5, so that the pixel point in the region to be adjusted may be considered to be in a semitransparent state.

In addition, when the pixel values of the pixel points in the region to be adjusted are adjusted, the pixel values of all the pixel points can be adjusted, and the pixel values of only part of the pixel points can be adjusted.

Based on the above situation, in an embodiment of the present invention, a pixel point belonging to a preset shape region in a region to be adjusted in an image frame to be processed may be determined, and then a pixel value of the determined pixel point in the image frame to be processed is adjusted to obtain a temporary image frame.

The preset shape region may be a square wave type region, a sawtooth type region, or the like.

Step S104B: and obtaining the image frame of which the pixel values of the pixels in the object region are the same as the pixel values of the pixels in the object region in the image frame to be processed before adjustment and the pixel values of the pixels in the non-object region are the same as the pixel values of the pixels in the non-object region in the temporary image frame according to the binary image, the image frame to be processed before pixel value adjustment and the temporary image frame.

In this step, when the pixel value of the image frame to be processed is adjusted, the pixel value of the pixel point with the same position as the pixel point of the non-object is adjusted, so that the pixel value of the pixel point is consistent with the pixel value of the pixel point in the temporary image frame, and the pixel value of the pixel point in the area where the object is located in the image frame to be processed can be kept and is not changed.

Because the pixel points in the binary image are divided into two types, one type is the pixel points belonging to the object, and the other type is the pixel points not belonging to the object, the pixel points with the same position as the non-object pixel points in the image frame to be processed are determined according to the pixel values of the pixel points in the binary image and the positions of the pixel points.

As can be seen from the foregoing steps S104A-S104B, in the scheme provided in this embodiment, the target region is not directly determined, but the target region is indirectly determined through a binary image, so that the pixel value of the pixel point in the target region is adjusted.

As can be seen from the above, in the scheme provided in this embodiment, a binary image is generated according to the determined pixel points belonging to the object in the image frame to be processed, and the binary image has the same size as the image frame to be processed and only includes two types of pixel points whose pixel values are the third preset pixel value and the fourth preset pixel value, one type is used for representing the object, and the other type is used for representing the content other than the object, so that the result of identifying the object in the image frame to be processed can be clearly and intuitively expressed through the binary image. The image frame with the adjusted pixel values is completed in two steps, firstly, the pixel values of the to-be-adjusted areas in the image frame to be processed are adjusted to obtain temporary image frames, then, according to the binary image and the corresponding relation of the pixel points between the image frame to be processed and the temporary image frames before the pixel value adjustment, the pixel values of the pixel points in the areas where the objects in the image frame to be processed are located are reserved, and the image frame with the adjusted pixels has the effect of visual extension.

The following describes the image frame after pixel adjustment with reference to a specific exemplary image.

Referring to fig. 3a-3d, four different cases of pixel value adjusted image frames are shown.

Specifically, fig. 3a shows an adjusted image obtained after transparency adjustment is performed on pixel points belonging to a square wave region in an edge region corresponding to the top pixel row and the bottom pixel row in an image frame to be processed.

Fig. 3b shows an adjusted image obtained after transparency adjustment is performed on pixels belonging to a square wave region in an edge region corresponding to the top pixel row and the bottom pixel row in an image frame to be processed, and transparency adjustment is performed on pixels in edge regions corresponding to the leftmost pixel column and the rightmost pixel column.

Fig. 3c shows an adjusted image obtained after adjusting the pixel values of the pixel points in the edge region corresponding to the top pixel row and the bottom pixel row in the image frame to be processed and the region to be adjusted determined along the height direction of the image frame to be processed to the pixel values corresponding to black.

Fig. 3d shows a processed image obtained after adjusting the pixel values of the pixel points in the edge area corresponding to the top pixel row, the bottom pixel row, the left pixel column and the right pixel column in the image to be processed and the area to be adjusted determined along the height direction of the image frame to be processed to the pixel values corresponding to black.

It can be seen from the above fig. 3a to 3D that the adjusted images can make the user feel stronger 3D effect.

In an embodiment of the present invention, when the third preset pixel value is 1 and the fourth preset pixel value is 0, the step S104B obtaining the image frame, in which the pixel value of the pixel point in the object region is the same as the pixel value of the pixel point in the object region in the image frame to be processed before the adjustment and the pixel value of the pixel point in the non-object region is the same as the pixel value of the pixel point in the non-object region in the temporary image frame, according to the binary image, the image frame to be processed before the adjustment of the pixel value and the temporary image frame includes the following steps S104B1-S104B 4.

Step S104B 1: and according to the pixel points, carrying out image multiplication calculation on the binary image and the image frame to be processed before the pixel value adjustment to obtain a first intermediate image.

Specifically, in this step, the binary image, the image frame to be processed before the pixel value adjustment and the first intermediate image have the same size, the image multiplication is performed to calculate that the pixel value of the pixel point in the binary image is multiplied by the pixel value of the pixel point at the same position in the image frame to be processed before the pixel value adjustment and the binary image, and the pixel value obtained by multiplying the two pixel values is the pixel value of the pixel point at the same position in the first intermediate image and the binary image.

Because the binary image only contains two types of pixel values, the pixel value of the object pixel point is 1, and the pixel value of the non-object pixel point is 0, in the obtained first intermediate image, the pixel value of the object pixel point is the pixel value of the pixel point with the same position as the pixel point belonging to the object in the image frame to be processed, and the pixel value of the non-object pixel point is 0.

Step S104B 2: and carrying out pixel value inversion operation on the binary image to obtain a second intermediate image.

After the inversion operation, the pixel value of the object pixel point in the second intermediate image is 0, and the pixel value of the non-object pixel point is 1.

Step S104B 3: and according to the pixel points, carrying out image multiplication calculation on the second intermediate image and the temporary image frame to obtain a third intermediate image.

In the same manner as the image multiplication calculation in step S104B1, in the third intermediate image obtained by performing the image multiplication calculation on the second intermediate image and the temporary image frame, the pixel value of the object pixel is 0, and the pixel value of the non-object pixel is the pixel value of the pixel at the same position in the image frame to be processed.

Step S104B 4: and according to the pixel points, carrying out image addition calculation on the first intermediate image and the third intermediate image to obtain an image frame.

In the step, the image addition is calculated by adding the pixel values of the pixel points in the first intermediate image and the pixel values of the pixel points in the third intermediate image at the same position as the first intermediate image, and the pixel value obtained by adding the two pixel values is the pixel value of the pixel point in the image frame at the same position as the first intermediate image.

The above steps S104B1-S104B4 may adjust the pixel values of the pixels in the image to be processed according to the following expression:

I1*P+I2*(1.0–P)

wherein I1 denotes an image frame to be processed before pixel value adjustment, I2 denotes a temporary image frame, and P denotes a binary image.

As can be seen from the above, in the scheme provided in this embodiment, the pixel values of the pixel points in the object region in the image frame to be processed can be retained, and in addition, the pixel values of the pixel points in other regions except the object region in the temporary image frame can be extracted. On the basis, the extracted pixel values are superposed on the image frame to be processed according to the positions of the pixels, so that the adjustment of the pixel values of the pixels in the image frame to be processed can be conveniently and efficiently realized.

In an embodiment of the present invention, referring to fig. 4, a flowchart of a third image processing method is provided, and compared with the foregoing embodiment shown in fig. 1, in this embodiment, the step S102 of performing object recognition on the image frame to be processed to determine the object region where the object is located in the image frame to be processed includes the following steps S102a2-S102E 2.

Step S102a 2: and carrying out multilayer convolution transformation on the image frame to be processed to obtain the image after convolution and the image characteristics of the image frame to be processed, which are extracted by each layer of convolution transformation.

The multi-layer convolution transformation of the image frame to be processed can be understood as that the image frame to be processed is subjected to convolution transformation layer by layer, and the result of each layer after the convolution transformation is used as the input of the next layer for the convolution transformation.

Step S102B 2: and determining first object pixel points belonging to the object in the convolved image.

The image features can be extracted by performing convolution transformation on the image, so that the image after convolution can be regarded as the feature image of the image frame to be processed, the region for representing the object features can be determined by analyzing the image content of the image after convolution transformation, and the pixel points of the determined region are used as the first object pixel points belonging to the object in the image after convolution.

Since whether the object is a human being, an animal, or another object such as a vehicle is generally set in advance, feature information of the object may be set in advance, and an area in the image after the convolution conversion, which matches the preset feature information, may be analyzed to specify an area for characterizing the feature of the object.

Step S102C 2: and performing multilayer deconvolution transformation on the convolved image based on the image characteristics to obtain the deconvoluted image.

Each convolution transformation may be understood as one image feature extraction, and therefore, when performing multi-layer convolution transformation on an image frame to be processed, the image features extracted after each layer of convolution transformation reflect edge information on different dimensions of the image frame to be processed. Therefore, on the basis of the image characteristics obtained by performing convolution transformation each time, multilayer deconvolution transformation is performed on the convolved image, so that the obtained deconvolved image has rich edge information.

Step S102D 2: and selecting second object pixel points corresponding to the first object pixel points in the deconvolved image.

Those skilled in the art will understand that, there is a corresponding relationship between the pixel point in the deconvolved image and the pixel point in the convolved image, and in this step, the pixel point determined in the step S102B2 is selected from the deconvolved image by using the corresponding relationship, and the selected pixel point is used as the second object pixel point.

Step S102E 2: determining a region including a third object pixel point in the image frame to be processed as an object region where an object is located, wherein the third object pixel point is as follows: and the positions of the pixel points in the image frame to be processed are the same as the positions of the second object pixel points.

After the multi-layer convolution transformation and the deconvolution transformation are performed on the image frame to be processed through the steps S102a2-S102D2, the pixel point in the region where the object is located, that is, the second object pixel point, is already identified in the obtained deconvolved image, and therefore, in the image frame to be processed, the region where the third object pixel point, which is the same as the second object pixel point in the deconvolved image, is located is the object region where the object is located.

As can be seen from the above, in the scheme provided in this embodiment, the features of the deep layers in the image frame to be processed are extracted from the convolved image obtained through the multi-layer convolution, the first object pixel point belonging to the object is determined in the convolved image obtained at the last time, and the first object pixel point belonging to the object in the convolved image can be more accurately determined based on the features of the deep layers in the image frame to be processed. In addition, when deconvolution processing is carried out, the image characteristics of the image frame to be processed, which are extracted by each layer of convolution operation, are introduced, so that the image after deconvolution can better recover the edge information of the image, and the object area where the object in the image frame to be processed is located can be more accurately determined.

In another embodiment of the present invention, the steps S102A-S102E shown in fig. 4 can be implemented by a neural network model. Specifically, referring to fig. 5, a schematic diagram of a convolutional neural network is shown.

In the diagram, the rectangular figures with different heights represent different feature maps of the image frame to be processed, the heights of the respective rectangular figures on the left side are gradually reduced, and the arrows on this side represent operations of convolution, pooling and the like for realizing step S102A. The convolution layers carry out convolution transformation and pooling treatment layer by layer according to the arrow direction of the diagram, and after the image frame to be processed is subjected to multilayer convolution transformation, a convolved image, namely the image represented by the black rectangular graph with the shortest middle position length in the diagram, is obtained. From the convolved image, a first object pixel point belonging to the object can be determined. In addition, in the schematic diagram, the height of each rectangular figure on the right side is gradually increased, and the arrow on this side indicates operations of deconvolution, deballation and the like for realizing step S102C. On the side, based on image features (shown as an upper broken line arrow in fig. 5) extracted by each layer of convolution transformation in the convolution transformation, the image after convolution is subjected to deconvolution transformation to obtain an image after deconvolution, so that the image after deconvolution has rich edge information, original edge information in the image to be processed is effectively reserved, and loss of the edge information in the convolution and deconvolution transformation processes is reduced.

The convolution kernel corresponding to each layer of convolution transformation may be a convolution kernel selected to reduce the image, for example, a non-filled convolution kernel having a size of 3x3 may be selected, or a convolution kernel having a size of 3x3 and a step size of 2 may be selected, and in addition, a nonlinear process implemented by a function such as ReLU may be added after each convolution transformation, so as to enhance the convergence rate of the convolutional neural network. In addition, the convolutional neural network model may further include a pooling layer group with a size of 2 × 2 and a step size of 2. In response, the deconvolution kernels of each layer of deconvolution transform may be selected such that the image is enlarged.

In this case, since the number of output images gradually increases as the convolution transform is performed, the number of images output by each layer of convolution transform gradually decreases, and the above-described flow for realizing convolution may also be referred to as a contraction path. Since the image output by each layer of deconvolution transform gradually increases as the deconvolution transform is performed, the above-described procedure for realizing deconvolution may also be referred to as an expansion path.

In one implementation, the number of layers of the convolutional layer may be 23.

Corresponding to the image processing method, the embodiment of the invention also provides an image processing device.

Referring to fig. 6, there is provided a schematic structural diagram of a first image processing apparatus, the apparatus including:

an image frame acquiring module 601, configured to acquire an image frame to be processed;

an object region determining module 602, configured to perform object identification on the image frame to be processed, and determine an object region where an object in the image frame to be processed is located;

a to-be-adjusted region determining module 603, configured to determine at least one to-be-adjusted region in the to-be-processed image frame;

the pixel adjusting module 604 is configured to determine a target region, where the target region belongs to the region to be adjusted and does not belong to the object region, and adjust a pixel value of a pixel point in the target region to obtain an image frame after the pixel value is adjusted.

In an embodiment of the present invention, the pixel adjusting module 604 may specifically adjust the pixel value of the pixel point in the target region according to at least one of the following manners:

the first method is as follows: adjusting the pixel value of a pixel point in the target area to be a first preset pixel value;

the second method comprises the following steps: reducing the value of the transparency component in the pixel value of the pixel point in the target area;

the third method comprises the following steps: performing blurring processing on the target area;

the method is as follows: adjusting the pixel value of the region edge pixel point to a second preset pixel value, wherein the region edge pixel point is as follows: and the pixel points in the image frame are positioned at the boundary of the target area and the non-target area.

In an embodiment of the present invention, the module 603 for determining an area to be adjusted includes:

the first area determination sub-module 603A: for determining at least one region to be adjusted in an edge region of the image frame to be processed.

In another embodiment of the present invention, the first area determining sub-module 603A is specifically configured to:

the second region determination sub-module 603B: for determining at least one area to be adjusted in the width direction and/or the height direction of the image frame to be processed.

In one implementation, the to-be-adjusted region determining module 603 includes a first region determining submodule 603A and a second region determining submodule 603B.

In an embodiment of the present invention, referring to fig. 7, a schematic structural diagram of a second image processing apparatus is provided, and compared with the foregoing embodiment shown in fig. 6, in this embodiment, the object region determining module 602 includes:

the object identification submodule 602A: the image processing device is used for carrying out object identification on the image frame to be processed and determining pixel points belonging to an object in the image frame to be processed;

the binary image generation sub-module 602B: the image processing device is used for generating a binary image which has the same size with the image frame to be processed, the pixel value of an object pixel point is a third preset pixel value, and the pixel value of a non-object pixel point is a fourth preset pixel value, wherein the object pixel point is as follows: the positions of the pixel points are the same as those of the pixel points belonging to the object in the image frame to be processed, and the non-object pixel points are as follows: pixels other than the object pixel;

in this embodiment, the pixel adjusting module 604 includes:

the first image frame acquisition sub-module 604A: the image processing device is used for adjusting pixel values of pixel points in the region to be adjusted in the image frame to be processed to obtain a temporary image frame;

the second image frame obtaining sub-module 604B: and the image frame is used for obtaining the image frame of which the pixel values of the pixels in the object region are the same as the pixel values of the pixels in the object region in the image frame to be processed before adjustment and the pixel values of the pixels in the non-object region are the same as the pixel values of the pixels in the non-object region in the temporary image frame according to the binary image, the image frame to be processed before adjustment of the pixel values and the temporary image frame.

As can be seen from the above, in the scheme provided in this embodiment, a binary image is generated according to the determined pixel points belonging to the object in the image frame to be processed, and the binary image has the same size as the image frame to be processed and only includes two types of pixel points whose pixel values are the third preset pixel value and the fourth preset pixel value, one type is used for representing the object, and the other type is used for representing the content other than the object, so that the result of identifying the object in the image frame to be processed can be clearly and intuitively expressed through the binary image. The image frame with the adjusted pixel values is completed in two steps, firstly, the pixel values of the to-be-adjusted areas in the image frame to be processed are adjusted to obtain temporary image frames, then, according to the binary image, the pixel point corresponding relation between the image frame to be processed before the pixel value adjustment and the temporary image frames, the pixel values of the pixel points in the to-be-processed image frames are adjusted, the pixel values of the pixel points in the areas where the objects in the image frame to be processed are located are reserved, and the image frame with the adjusted pixels has the effect of visual extension.

In an embodiment of the present invention, under the condition that the third preset pixel value is 1 and the fourth preset pixel value is 0, the second image frame obtaining sub-module 604B is specifically configured to:

In an embodiment of the present invention, the object region determining module 602 is specifically configured to:

An embodiment of the present invention further provides an electronic device, as shown in fig. 8, which includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, where the processor 801, the communication interface 802, and the memory 803 complete mutual communication through the communication bus 804,

a memory 803 for storing a computer program;

the processor 801 is configured to implement the following steps when executing the program stored in the memory 803:

acquiring an image frame to be processed;

Besides, the electronic device may also implement other image processing methods as described in the foregoing embodiments, and details are not described here.

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.

In still another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the image processing method described in any of the above embodiments.

In yet another embodiment, the present invention further provides a computer program product containing instructions which, when run on a computer, cause the computer to perform the image processing method described in any of the above embodiments.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, the electronic device, the computer-readable storage medium, and the computer program product embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and in relation to them, reference may be made to the partial description of the method embodiments. The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. An image processing method, characterized in that the method comprises:

acquiring an image frame to be processed;

2. The method according to claim 1, wherein the determining at least one region to be adjusted in the image frame to be processed comprises:

and/or

3. The method according to claim 2, wherein the determining at least one region to be adjusted in the edge region of the image frame to be processed comprises:

4. The method according to any one of claims 1-3, wherein the adjusting the pixel values of the pixel points in the target region to obtain the image frame with the adjusted pixel values comprises:

performing blurring processing on the target area;

5. The method according to any one of claims 1-3, wherein the performing object recognition on the image frame to be processed and determining an object region where an object is located in the image frame to be processed comprises:

6. The method according to claim 5, wherein in a case that the third preset pixel value is 1 and the fourth preset pixel value is 0, obtaining, from the binary image, the image frame to be processed before pixel value adjustment and the temporary image frame, an image frame in which a pixel value of a pixel point in an object region is the same as a pixel value of a pixel point in an object region in the image frame to be processed before adjustment and a pixel value of a pixel point in a non-object region is the same as a pixel value of a pixel point in a non-object region in the temporary image frame includes:

7. The method according to any one of claims 1-3, wherein the performing object recognition on the image frame to be processed and determining an object region where an object is located in the image frame to be processed comprises:

8. An image processing apparatus, characterized in that the apparatus comprises:

9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.

10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.