RU2791810C2 - Method, equipment and device for image processing - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: essence of the invention lies in the fact that the target area and the background area are determined in the image by performing mask segmentation for this image. Different colour processing modes are applied to these target area and background area, so that the brightness of the target area becomes larger than the brightness of the background area, or the colour saturation in the target area becomes higher than the colour saturation in the background area, and the main object corresponding to the target area is more highlighted.

EFFECT: providing the possibility of improving the quality of the resulting image.

37 cl, 10 dwg

Description

The technical field to which the invention belongs

The present invention relates to the field of technologies used in terminals, and in particular, to a method, apparatus and apparatus for image processing.

State of the art

Photography is the use of a photo or video camera or video recorder to record an image of a person or object. For different scenes, different photography methods are applied, such as night photography, rain photography, building photography, and portrait photography. Dynamic photography for shooting moving objects is also a type of photography, but must comply with a specific, special principle. As science advanced and developed, photography became easier and more popular.

With the expansion of the bandwidth of the communication network and the increase in the processing capacity of the terminal, taking pictures and sharing videos and images has become more convenient, and video consumption has become a new way of life for all people. Video has quickly become a traffic-intensive service on a communications network, and video traffic is expected to account for 80% to 90% of total network traffic over the next few years.

In daily life, photography has become the main way for people to show themselves and bring out the beauty of things. People want to take pictures in more interesting ways. For example, when photographing, special image or video processing effects are applied to achieve a “what you see is what you get” experience. Therefore, for the layman, more and more new image processing technologies need to be integrated into the terminal.

At present, the video recording function in the terminal equipment is monotonous. Today, only ordinary video photography can be carried out, without some personalized effects.

The essence of the invention

The present invention provides an image processing method. The target area and the background area are determined on the image by performing image segmentation using a mask (template). Different color processing modes are applied to these target area and background area, so that the brightness of the target area becomes larger than the brightness of the background area, or the color saturation in the target area becomes larger than the color saturation in the background area, so that the main object corresponding to the target area is more highlighted. . This allows the user to have special moving image effects in the process of taking a photo or a video of taking a photo (movie shooting) and improve the result of taking a photo for the user.

In the embodiments of the present invention, the following technical solutions are proposed.

According to a first aspect, one embodiment of the present invention provides an image processing method. This method is applied to the video recording procedure and includes: taking N images; and performing step 1 and step 2 on each of the N images to obtain the target video:

step 1: determining the target area and the background area of the image based on the category of the object in the image, where the background area is an image area different from the target area; And

step 2: processing the target area and the background area in different color processing modes to obtain a target image where the color saturation in the target area in the target image is higher than the color saturation in the background area in this target image, or the brightness of the target area in the target image is higher than the brightness of the background area in this target image.

According to a second aspect, one embodiment of the present invention provides an image processing apparatus. This apparatus is applied to a video photographing procedure and includes: a photographic module configured to take N images; a decision module configured to determine a target area and a background area in each of the N images based on the category of an object in that image, where the background area is an image area different from the target area; and a color processing unit configured to process the target area and the background area in different color processing modes to obtain a target image, where the color saturation in the target area in the target image is greater than the color saturation in the background area in the target image, or the brightness of the target area in the target image more brightness of the background area in the target image.

According to a third aspect, one embodiment of the present invention provides an image processing method. This method applies to the process of photographing and contains:

getting an image; determining a target area and a background area in the image based on the category of the object present in the image, where the background area is an area of the image other than the target area; and processing the found target area and background area using different color processing modes to obtain a target image, where the color saturation in the target area of the target image is greater than the color saturation in the background area of this target image, or the brightness of the target area of the target image is greater than the brightness of the background area of this target image .

According to a fourth aspect, one embodiment of the present invention provides an image processing apparatus. The equipment contains:

a photographic module configured to acquire an image; a decision module configured to determine the target area and the background area of the considered image based on the category of the object in this image, where the background area is an image area different from the target area; and a color processing unit configured to process the target area and the background area using different color processing modes to obtain a target image, where the color saturation in the target area of the target image is greater than the color saturation in the background area of this target image, or the brightness of the target area of the target image is greater than the brightness the background area of this target image.

According to the first aspect, the second aspect, the third aspect or the fourth aspect in one of the possible options, the procedure for determining the target area and the background area in the image based on the category of the object in this image contains: performing semantic segmentation of the image to obtain k masks, where these k masks correspond to various categories of objects; And

if k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area;

if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, determine the image area corresponding to these k0 masks for the main objects as the target area, and determine the image area corresponding to one from the remaining masks, as a background region, where k0 is a non-negative integer less than k;

if k is greater than 2, determine, as the target area, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determine the area of the image corresponding to the rest of the masks, as the background area;

if k is greater than 2, determining a target mask from a plurality of k masks based on predetermined object category priorities, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the rest of the masks as a background area; or

if k is greater than 2, determining a target mask from the set of k masks in accordance with a selection command from the user; and determining an image area corresponding to the target mask as the target area, and determining an image area corresponding to the rest of the masks as the background area. This method can be specifically performed by the solver.

According to the first aspect, the second aspect, the third aspect, or the fourth aspect in one possible embodiment, the procedure for processing the target area and the background area using different color processing modes includes, but is not limited to:

preserving the color of the target area, and performing black and white processing of the background area;

maintaining the color of the target area, and performing darkening processing of the background area;

preserving the color of the target area, and performing "blurring" processing of the background area;

enhancing the color of the target area, and performing black and white processing of the background area;

enhancing the color of the target area, and performing darkening processing of the background area; or

enhancing the color of the target area, and performing a "blurring" treatment on the background area.

In addition, the background area processing method may further comprise a dark tone filtering method such as a retro filtering method or a film filtering method. This method, in particular, is carried out by the color processing module.

According to the first aspect, the second aspect, the third aspect or the fourth aspect in one of the options, the category of the object may be at least one of the list - a person, an animal, vegetation, a given object or background.

More specifically, in the technical implementation options mentioned above, the processor may call a program and instruction from a storage device to perform appropriate data processing, such as implementing algorithms and/or receiving signals.

According to a fifth aspect, one embodiment of the present invention provides an image processing method. This method is applied to the video recording procedure and, in particular, contains: determining the category of the object included in each received image; if in the first N0 images relative to the current image, the number of groups of neighboring images with the same object, defined as different categories of objects, is not more than a given threshold value, the determination of the target area and the background area in the current image based on the category of the specified object in the image, where the background the area is the area of the current invention, different from the target area; and processing the target area and the background area in the first color processing mode and the second color processing mode, respectively, to obtain a target image, where the color saturation in the target area of the target image is greater than the color saturation in the background area of this target image, or the brightness of the target area of the target image more brightness of the background area of this target image; or if in the first N0 images of the current shooting (current image), the number of groups of neighboring images with the same object, defined as different categories of objects, is greater than a predetermined threshold, processing this current image in the first color processing mode or in the second color processing mode to get the target image.

According to a sixth aspect, one embodiment of the present invention provides an image processing apparatus. This image processing apparatus is used in a video recording process and specifically comprises: a photographic module configured to acquire an image; a decision module configured to determine the category of the object included in each received image; an evaluation module configured to: if in the first N0 images relative to the current image, the number of groups of neighboring images with the same object, defined as different categories of objects, is not greater than a given threshold, control the decision module to determine the target area and the background area in the current image based on the category of the specified object in the image, where the background area is an area of the current image, different from the target area; and controlling the color processing unit to process the target area and the background area in the first color processing mode and the second color processing mode, respectively, to obtain a target image where the color saturation in the target area of the target image becomes greater than the color saturation in the background area of this target image, or the brightness of the target area of the target image becomes greater than the brightness of the background area of this target image; or if in the first N0 images relative to the current image, the number of groups of neighboring images with the same object, defined as different categories of objects, is greater than a predetermined threshold, control the flicker elimination unit to process the current image in the first color processing mode or in the second mode color processing to obtain the target image.

It should be understood that the current video image can be considered as an image being recorded at some point in time. In some scenarios, this moment may be considered as some generalized moment, or in some other scenarios, it may be a specific moment, for example, the most recent moment, or the moment in which the user is interested.

According to the fifth aspect or the sixth aspect, in one of the possible options, the procedure for determining the target area and the background area in the current image based on the category of the object in this image contains: performing semantic segmentation of the image in order to obtain k masks, where these k masks correspond to different categories of objects ; And

if k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area;

if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than the predetermined threshold value, the image area corresponding to these k0 masks for the main objects is determined as the target area, and the image area corresponding to one from the remaining masks, as a background region, where k0 is a non-negative integer less than k;

if k is greater than 2, determine, as the target area, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determine the area of the image corresponding to the rest of the masks, as the background area;

if k is greater than 2, determining a target mask from a plurality of k masks based on predetermined object category priorities, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the rest of the masks as a background area; or

if k is greater than 2, determining the target mask from the set of k masks in accordance with a selection command from the user; and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the rest of the masks as a background area. This method can in particular be carried out by a solver.

According to the fifth aspect or the sixth aspect, in one possible embodiment, the procedure for processing the target area and the background area in different color processing modes includes, but is not limited to:

preserving the color of the target area, and performing black and white processing of the background area;

maintaining the color of the target area, and performing darkening processing of the background area;

preserving the color of the target area, and performing "blurring" processing of the background area;

enhancing the color of the target area, and performing black and white processing of the background area;

enhancing the color of the target area, and performing darkening processing of the background area; or

enhancing the color of the target area, and performing a "blurring" treatment on the background area.

In addition, the background area processing method may further comprise a dark tone filtering method such as a retro filtering method or a film filtering method. This method, in particular, is carried out by the color processing module.

According to the fifth aspect or the sixth aspect, in one of the possible options, the category of the object may be at least one of the list - a person, an animal, vegetation, a given object or background.

More specifically, in the technical implementation options listed above, in the above technical implementation options, the processor can call a program and instruction from a storage device to perform appropriate data processing, such as implementing algorithms and/or receiving signals.

According to a seventh aspect, one embodiment of the present invention provides a terminal including a video camera, a storage device, a processor, and a bus. These video camera, storage device and processor are connected to each other via a bus. The video camera is configured to capture an image and the storage device is configured to store the computer program and instructions. The processor is configured to call the computer program, instructions, and the captured image stored in the storage device, and further specifically configured to allow the terminal to perform any one of the above options.

According to the seventh aspect, in one possible embodiment, this terminal further comprises an antenna system. This antenna system receives/transmits a radio communication signal under the control of the processor for the purpose of radio communication with a mobile communication network. The mobile communication network may be one or more of the following networks: GSM network, CDMA network, 3G network, 4G network, 5G network, FDMA network, TDMA network, PDC network, TACS network, AMPS network, WCDMA network, TDSCDMA network, Wi-Fi and LTE network.

The technical solutions of the above possible options can be combined without conflicting with the nature of the present invention.

When using conventional technology when photographing video or images, the special effects are not rich enough without distinguishing between individual objects or colors in any image. According to the present invention, different areas of an image can be distinguished from one another using colors, so that the special effects of a photo or video are enhanced, so that the main subject and target can be better distinguished in the image. Thus, the main roles become more visible.

Brief description of the drawings

Fig. 1 is a simplified diagram of the structure of a terminal according to one embodiment of the present invention;

Fig. 2 is a logic diagram of an image processing method according to one embodiment of the present invention;

Fig. 3 shows an example of a mask identifier according to one embodiment of the present invention;

Fig. 4 shows another example of a mask identifier according to one embodiment of the present invention;

Fig. 5 shows a simplified scheme for determining a target mask according to one embodiment of the present invention;

Fig. 6 shows another simplified scheme for determining the target mask according to one embodiment of the present invention;

Fig. 7 shows another simplified scheme for determining the target mask according to one embodiment of the present invention;

Fig. 8 shows another simplified scheme for determining the target mask according to one embodiment of the present invention;

Fig. 9 is a simplified diagram of an image processing apparatus according to one embodiment of the present invention; And

Fig. 10 is another simplified diagram of an image processing apparatus according to an embodiment of the present invention.

Description of options

The following clearly and completely describes the technical solutions of the present invention with reference to the accompanying drawings of these embodiments of the present invention. Obviously, the described options represent only some, but not all, options for the present invention. All other variants obtained even by ordinary specialists in the considered area on the basis of the variants of the present invention presented here without any special creative efforts, fall within the protection scope of the present invention.

In one embodiment of the present invention, the terminal may be a device that provides the user with photo and video capabilities and/or a data connection, a hand-held device having a wireless (radio) connection function, or another processing device connected to a wireless (radio) modem. , such as a digital video camera, a single lens reflex camera, a mobile phone (or also called a "cell phone"), or a smartphone. The terminal may be a handheld, handheld, handheld or wearable device (e.g., smart wristwatch), tablet computer, personal computer (PC), personal digital assistant (PDA (Personal Digital Assistant)), car computer, drone ( unmanned aerial vehicle), aircraft or other similar object.

Fig. 1 is a simplified diagram of one of the options for the hardware structure of the terminal 100.

As shown in FIG. 1, terminal 100 may include components such as a RF module 110, a storage device 120, an input module 130, a display module 140, a video camera 150, an audio circuit 160 (comprising a speaker 161 and a microphone 162), a processor 170, an external interface 180, and a source 190. nutrition. One skilled in the art will appreciate that in FIG. 1 is just an example of an intelligent terminal or multifunctional device, which does not impose any restrictions on the intelligent terminal or multifunctional device. Such an intelligent terminal or multifunctional device may contain more or fewer components than shown in the drawing, or combinations of such components, or contain other components. For example, such an intelligent terminal or multifunctional device includes at least a storage device 120, a processor 170, and a video camera 150.

The video camera 150 is configured to capture an image or video and can be operated in accordance with a command from an application program to perform a photographing function or a photographing (shooting) video function. The camcorder may include components such as a video lens, a light filter, and an imager. Light rays emitted or reflected by an object enter the lens, pass through a filter, and eventually converge on the imager. The lens is configured primarily to converge, into an image, the light emitted or reflected by all objects (which may also be referred to as the scene to be photographed, the objects to be photographed, the target scenario or targets, and may also be considered an image of the scenario that the user wishes to photograph) within the angle of view for photographing. The color filter is mainly configured to filter out excess light waves (eg, light waves other than visible light, such as infrared radiation) from the light beams. The imager is mainly configured to: perform optoelectrical conversion of the received optical signal, convert the optical signal into an electrical signal, and input the resulting electrical signal to the processor 170 for further processing. The video camera can be located on the front side of the terminal or on the back side of the terminal. The specific number and specific location of cameras can be determined flexibly based on designer requirements or supplier policy. In the present application, this is not limited in any way.

The input module 130 may be configured to receive numeric or character input and generate a key input related to user settings and control of the functions of the multifunction equipment. In particular, the input module 130 may include a touch screen 131 and/or other input device 132. The touch screen 131 can collect a touch operation (eg, an operation performed by a user on or near the touch screen using some suitable object such as a finger, a ball joint, or a stylus) and control the operation of the associated communication equipment based on a predetermined program. The touch screen may detect a user's touch on the touch screen, convert that touch into a corresponding touch signal, transmit that touch signal to processor 170, and may also receive and execute a command transmitted by processor 170. The touch signal contains at least information about the coordinates of the contact point. This touch screen 131 can create an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen may be implemented in various types of devices such as resistive type, capacitive type, infrared type, or surface acoustic wave screens. In addition to the touch screen 131, the input module 130 may also include another input device. In particular, this other input device 132 may be, without limitation, one or more of the following devices - a physical keyboard, a function key (for example, a volume key and/or an on/off key 133), a trackball, a mouse, a joystick, or other similar device.

The display module 140 may be configured to display user input or user information, various menus used by the terminal 100, an interactive interface, a file, and/or playback of any media file. In this embodiment of the present invention, the display module is further configured to present an image/video obtained by the terminal using the video camera 150. This image/video may be a preview image/video in some photographic modes, a photographed original image/video, and a target image/video. to which processing was applied in accordance with a special algorithm after photographing.

Further, the touch screen 131 may cover the display panel 141 . Upon detecting a touch operation on or near the surface of the touch screen 131, the touch screen 131 passes the detected touch operation to the processor 170 to determine the type of touch event. The processor 170 then generates an appropriate visual output on the display panel 141 based on the detected touch event type. In this embodiment, the touch screen and the display module can be integrated in one component to implement the input, output, and display functions of the terminal 100. For ease of description, in this embodiment of the present invention, the display touch screen represents a plurality of touch screen and display module functions. In some embodiments, the touch screen and the display module may alternatively be used as two independent components.

Memory 120 may be configured to store commands and data. This storage device 120 may mainly include an instruction storage area and a data storage area. The data storage area can store data such as media file and text. The instruction storage area may contain software blocks such as an operating system, an application, and an instruction required for at least one function, a subset of functions, or an extended set of functions. Memory 120 may further comprise non-volatile random access memory and provide processor 170 with a number of functions including, but not limited to, control of hardware, software, and data resources in the computer processing device and support for software and application control. The storage device 120 is further configured to store the media file and store the executable program and application.

The processor 170 is the control center of the terminal 100 and is connected to various parts of the entire terminal through various interfaces and lines. This processor 170 performs various functions of the terminal 100 and processes data by executing instructions stored in the memory 120 and retrieving data stored in the memory 120 to perform general control of the terminal. Optionally, processor 170 may include one or more processor modules. Preferably, this processor 170 may be integrated with an application processor and a modem processor. The application processor mainly executes an operating system, performs user interface functions, executes an application program, and the like. The modem processor mainly handles the radio functions. It can be understood that, in the alternative, the modem processor may not be integrated into the processor 170. In some embodiments, the processor and storage device may alternatively be implemented on the same integrated circuit chip. In some embodiments, the processor and storage device may, respectively, be implemented on independent integrated circuit chips. Processor 170 may be further configured to: generate an appropriate operation control signal, transmit that operation control signal to an appropriate component in a computer processing device, and read and process data by software, and in particular read and process data and programs in storage device 120 so that the functional modules in the processor 170 perform the appropriate functions to control the respective components to perform actions as required by the instructions.

RF module 110 may be configured to receive and transmit information, or to receive and transmit a signal during a call. For example, the RF module 110 receives downlink information from the base station, then transmits this downlink information to the processor 170 for processing, and transmits the corresponding uplink data to the address of the base station. Typically, the RF module includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a directional coupler (coupler), a low noise amplifier (LNA (Low Noise Amplifier, LNA)), a duplexer, and other such components. In addition, the RF module 110 can further communicate with the network device and with the other device via radio. Such radio communication may use any communication standard or protocol, including, without limitation, the Global System for Mobile Communications (GSM) standard, the General Packet Radio Service (GPRS) standard, the multiple access standard with Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Message Service, SMS) and other similar standards.

Audio circuit 160, speaker 161, and microphone 162 can provide an audio interface between the user and terminal 100. Audio circuit 160 can provide speaker 161 with an electrical signal converted from the received audio data, and speaker 161 converts this electrical signal into an audio signal to emit sound. In addition, the microphone 162 is configured to collect an audio signal and can further convert the collected audio signal into an electrical signal. The audio circuit 160 receives this electrical signal, converts the electrical signal into audio data, transmits the audio data to the processor r 170 for processing, and then transmits the processed audio data to, for example, another terminal via the RF module 110, or transmits the processed audio data to the storage device 120 for further processing. The audio circuit may include a headphone (headphone) jack 163 configured to provide a connection interface between the audio circuit and the headphones. The specific number and specific layouts of loudspeakers and microphones can be flexibly determined based on designer requirements or supplier policy. In the present application, this is not limited in any way.

Terminal 100 further includes a power source (eg, battery) 190 that powers each component. Preferably, this power supply may be logically connected to the processor 170 using a power management system to implement functions such as charge, discharge, and power management using this power management system.

Terminal 100 further includes an external interface 180. This external interface may be a standard micro USB port, or may be a multi-pin connector. This external interface can be configured to connect terminal 100 to other communications equipment, or can be configured to connect to a charger to charge the battery of terminal 100.

Although not shown, terminal 100 may further include a flash lamp, a "wireless fidelity" (Wi-Fi) module, a Bluetooth module, sensors with various functions, and other such components. Details are not described here. To the terminal shown in Fig. 1, some or all of the methods described below may be applied.

The present invention can be applied to a terminal having a photographing function (including at least one of a normal photographing function or a video photographing function), so that the realized product may be in the form of an intelligent terminal, for example, it may be a product in which a video camera is installed. , such as a mobile phone, tablet, direct vision (DV) camera, camcorder, camera, laptop computer, laptop computer, intelligent robot, TV, security system or drone (unmanned aerial vehicle). In particular, the functional modules of the present invention may be deployed on a digital signal processor (DSP) integrated circuit chip of an appropriate device, and may in particular be an application program or software of a DSP chip. According to the present invention, these functional modules are deployed in the terminal to implement the image processing function by means of installed or updated software in cooperation with the corresponding hardware.

The present invention is mainly applied to a scenario in which photographing of an image or video is performed using a terminal. People have higher and higher requirements for video photography and expect to achieve a full range of special effects in video processing in the process of taking photos to achieve "what you see is what you get" video in photography. According to the present invention, segmentation of the main object can be performed on an image or video, and the colors of various areas can also be corrected to realize real-time image special effects.

See FIG. 2. In FIG. 2 is a flow diagram of an image processing method according to one embodiment of the present invention. This method can be carried out in the process of photographing or video. In the following, the present invention will be described using examples.

Example 1: Taking a photo of an image (photo mode)

For this example, during a specific implementation, the terminal may configure the photographing mode. In this photographing mode, the method may include the following steps.

Step 21: Taking (which can also be understood as photographing or capturing) an image.

In particular, when the user captures an image, the corresponding preview stream is also displayed on the display screen. The term "preview image" generally refers to one of the images in the preview stream. When the user presses the shutter release, a photographic image is obtained. The photographed image has a size such as but not limited to 1920 x 1080.

Step 22: Determining the target area and the background area in the image based on the content (which can be understood as scene semantics) of the photographed image. More specifically, a target area and a background area in an image may be determined based on the category of an object in that image. The background area is an area of the image that is different from the target area. The target area corresponds to the target object or target body in the image, namely, the object that the user expects to highlight (underline) in the image and which may be associated with an interactive user selection or with system settings. In particular, step 22 may include steps S221 to S224.

S221: Image pre-processing.

The photographed image of the original size is downsampled (subsampled) and converted to an image with a lower resolution. The amount of calculation can be reduced when calculations are performed based on a small image. During a particular implementation, the original size (eg, m0 x n0) may be downsampled to a size of m x n. Smaller values of m and n indicate a smaller amount of subsequent calculations. However, if the m and n values are too small, the pixel resolution is subsequently reduced. Experiments have shown that suitable ranges for m and n are [128, 512], and more specifically [256, 300]. In addition, the values m and n may or may not be equal to one another. For example, a 1920 x 1080 image can be downsampled to 256 x 256.

S222: Inputting a downsampled m x n image into a neural network to perform semantic segmentation to determine an image mask (Mask).

Semantic segmentation is understood as segmentation at the pixel level, carried out in relation to an object in the image, and the category of the object is marked for each pixel. An area that is not labeled with an object category is labeled as "background".

In particular, semantic segmentation can use a deep learning algorithm based on a convolutional neural network (CNN (Convolutional Neural Network)). In particular, the CNN-based network model is described as follows:

x mz, и n = 2n1 = 4n2 =, ..., =

x nz. Величина z и соотношение кратности могут быть определены на основе характеристик алгоритма и проектных требований.(1) Downsampling and convolution are performed on an mxn image to obtain an m1 x n1 image, an m2 x n2 image, ..., and an mz x nz image, and then extract the semantic features of the image layer by layer to obtain an m1 x feature map n1, feature maps m2 x n2, ..., and feature maps mz x nz, namely multi-level semantic features, where m1, m2, ..., and mz are in a multiple ratio with the number m and less than this number, and where n1, n2, ..., nz are in multiple ratio with the number n and less than this number. For example, m = 2m1 = 4m2 =, ..., =

x mz, and n = 2n1 = 4n2 =, ..., =

x nz. The z value and the multiplicity ratio can be determined based on the characteristics of the algorithm and the design requirements.

(2) The m1 x n1 feature map, the m2 x n2 feature map, ..., and the mz x nz feature map are upsampled (resampled) and convolved to merge the multilevel semantic features.

The convolution, downsampling, and upsampling methods mentioned above may use techniques that are well known in the industry and are not limited to those listed herein.

(3) The object category to be identified in the image is determined, a score is calculated for each object category at each pixel, the object category (which may be called “category” for short) with the highest score is used as the classification result for the pixel under consideration, and finally As a result, a mask graph is obtained, namely a mask.

For example, if the terminal can identify k categories of objects (eg, at least one category from a list - person, animal, vegetation, other specified object, or background), k images can be obtained. Each pixel in the image receives a score that belongs to some category of the object. A higher score indicates a higher probability that the pixel belongs to the corresponding object category.

Once the object category for a pixel is determined, that pixel can be identified. For example, 1 is used to represent a person, 2 is used to represent a vehicle (car), 3 is used to represent an animal, 4 is used to represent vegetation, and 0 is used to represent a background. This is just an example and does not constitute any limitation. The user can randomly set the number of categories, the categories themselves, and the identification method based on design requirements. A specific example can be shown in FIG. 3. The neural network classifies all pixel areas where the car is located as a car and assigns them an ID of 1. All the pixel areas in the surrounding background part are classified by the neural network as backgrounds and assigns them an ID of 0. For another example, in the mask passed by the neural network to the exit, areas of objects of the same category have the same marking. For example, the background marker is 0, the cat marker is 1, and the skateboard marker is 2. In the mask shown in FIG. 4, the same color can be further used to represent object markers of the same category. For example, a person, a horse, and a background are respectively identified using different colors.

The mask is the result of the semantic segmentation algorithm. In an image, all pixels belonging to any object category are labeled with a color or ID, and the background is also labeled with a color or ID. The image obtained after processing is called a mask, so that the segmentation result is presented on the display intuitively.

Image content can contain a main subject and a background. For ease of description, respectively, the image mask may be a mask for the main object or a background mask. The mask for the main object may correspond to the main object identified using the segmentation method, including individual objects that the user expects to see highlighted ("underlined") in the image or in the photographed image, for example, a person, animal, vegetation, or a specific object (cup, table , clothes, decorations ...). The background mask corresponds to another area in the image that is not identified as a mask for the main object. The image mask corresponds to the full image. The ability to identify the mask for the main object is related to the characteristics of the neural network. For example, some neural networks can only identify a person and a background. Some neural networks can identify a person, a car, and a background. Some neural networks can identify a person, an animal, and a background. Some neural networks can only identify the animal and the background. Some neural networks can identify animal, vegetation and background...

It should be understood that the image may alternatively contain only the main subject, or may contain only the background. When the image contains only the main object, this main object can be identified as the background. These settings regarding the main subject and background in the image can be flexibly constructed and defined by the designer.

To train a deep neural network, a large amount of segmentation training data may be required, where the training data set contains a large number of images, including object segmentation categories, input images, and mask graphs. The set of tuning data can cover a variety of typical application scenarios for object segmentation and contain a variety of data. The input images and mask graphs in the training data set are used to train (tuning) the network in order to obtain an excellent network parameter, in other words, to obtain segmentation characteristics that satisfy the user. The resulting network parameter is used as the final computational parameter for the neural network.

S223: Determination of the target mask based on the plurality of masks.

For different images and neural networks with different capabilities, different masks can be obtained. The terminal may then determine from the set of masks one mask corresponding to the object that most needs to be highlighted and made visible on the display. In other words, you need to define the target mask. The procedure for determining the target mask includes, but is not limited to, the following several methods.

Method 1: If the mask collection contains only one mask for the main object and one background mask, the mask for the main object is determined as the target mask.

In particular, suppose that semantic segmentation is performed on the image to obtain k masks. These k masks correspond to different categories of objects. If k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area;

As shown in FIG. 5, the set of masks for the image at the output of the neural network contains only one mask A1 for the main object and one background mask. In this case, mask A1 can be defined as the target mask.

Method 2: When the mask pool contains multiple main object masks and one background mask, if the number of pixels included in any main object mask is greater than a specific threshold, that main object mask is determined as the target main object mask; or if the number of pixels included in any mask for the main object is less than a specific threshold value, change the labeling of this mask for the main object and also mark it as a background. The number of pixels included in the mask for the main object may be the number of pixels included in the area of the individual object in the image.

In particular, suppose that semantic segmentation is performed on the image to obtain k masks. These k masks correspond to different categories of objects. If k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of k masks is greater than a predetermined threshold value, the image area corresponding to these k0 masks for the main objects is determined as the target area, and the image area corresponding to the remaining masks, is defined as the background region, where k0 is a non-negative integer less than k.

As shown in FIG. 6, the set of masks for the output image of the neural network contains a mask A1 for the main object, a mask A2 for the main object, and a background mask. If the number of pixels included in the mask A1 is greater than the predetermined threshold and the number of pixels included in the mask A2 is not greater than the predetermined threshold, the mask A1 is determined as the target mask, and the marking of the mask A2 for the main object is changed and re-marked as background mask. The relabeled mask can be shown in FIG. 5. If the number of pixels included in the mask A1 is greater than the predetermined threshold and the number of pixels included in the mask A2 is also greater than the predetermined threshold, both masks A1 and A2 are determined as target masks. If neither the number of pixels included in mask A1 nor the number of pixels included in mask A2 is greater than a predetermined threshold value, these masks A1 and A2 are re-identified as background masks. In other words, the image in question does not contain a mask for the main object.

It should be understood that in any particular implementation procedure, both masks A1 and A2 may be assigned to the same object category or to different object categories.

Method 3: When the mask collection contains multiple main object masks and one background mask, the main object mask containing the largest number of pixels is selected as the target mask, the labeling of the other main object masks is changed and re-labeled as background masks.

In particular, suppose that semantic segmentation is performed on the image to obtain k masks. These k masks correspond to different categories of objects. If k is greater than 2, the image area corresponding to the mask for the main object containing the largest number of pixels and included in the set of k masks is determined as the target area, and the image area corresponding to the rest of the masks is determined as the background area.

As shown in FIG. 6, the set of masks for the output of the neural network contains a mask A1 for the main object, a mask A2 for the main object, and a background mask. The mask A1 containing the largest number of pixels is determined as the target mask, and the labeling of the mask A2 for the main object is changed and re-labeled as the background mask. This newly labeled mask can be shown in FIG. 5.

It should be understood that in any particular implementation procedure, both masks A1 and A2 may be assigned to the same object category or to different object categories.

Method 4: When the mask collection contains several main object masks and one background mask, and the several main object masks include masks for several object categories, the target mask is determined based on the priorities of these object categories. For example, if the priority of the human mask is higher than the priority of the car mask, the human mask is the target mask, and the marking of the car mask can be changed and this mask can be re-tagged as a background. For example, if the priority of a human mask is higher than the priority of an animal mask and higher than the priority of a vegetation mask, and the system has set priorities in such a way that all masks with priorities higher than the vegetation mask priority are masks for the main objects, both masks and the human mask, and an animal mask are considered target masks, and the marking of the vegetation mask can be changed and this mask can be re-labelled as a background. It should be understood that one or more individual objects may belong to the same category of objects.

In particular, suppose that semantic segmentation is performed on the image to obtain k masks. These k masks correspond to different categories of objects. If k is greater than 2, the target mask is determined from these k masks based on the given object category priorities. The area of the image corresponding to the target mask is defined as the target area, and the area of the image corresponding to the rest of the masks is defined as the background area.

As shown in FIG. 7, the set of masks for the output image of the neural network contains a mask A1 for the main object, a mask B1 for the main object, and a background mask. Masks A1 and B1 belong to different categories of objects, and the priority of mask A1 is higher than the priority of mask B1. If the system determines that any mask for the main object whose priority is not lower than the priority of mask B1 can be used as the target mask, both masks A1 and B1 are target masks. If the system determines that the mask for the main object, whose priority is higher than the priority of mask B1, can be used as the target mask, the mask A1 is determined as the target mask, and the labeling of the mask B1 is changed and re-labeled this mask as the background mask.

Method 5: If the mask collection contains multiple masks for the main objects and one background mask, the target mask can be determined according to the selection operation entered by the user. The input mode includes, but is not limited to, a selection command such as a screen touch or a voice command. The mask for the main object corresponding to the individual object selected by the user is the target mask.

In particular, suppose that semantic segmentation is performed on the image to obtain k masks. These k masks correspond to different categories of objects. If k is greater than 2, the target mask is determined from the plurality of k masks according to the selection command from the user. The area of the image corresponding to the target mask is defined as the target area, and the area of the image corresponding to the rest of the masks is defined as the background area.

As shown in FIG. 7, the set of masks for the output image of the neural network contains a mask A1 for the main object, a mask B1 for the main object, and a background mask. If the user presses, in the process of photographing, an individual object corresponding to the mask A1 on the touch screen, this mask A1 is determined as the target mask, and the labeling of the mask B1 is re-labeled as the background mask. If the user presses, in the process of photographing, an individual object corresponding to the mask B1 on the touch screen, this mask B1 is determined as the target mask, and the marking of the mask A1 is re-marked as the background mask.

Method 6: If the mask collection contains multiple masks for main objects and one background mask, and the multiple masks for main objects include masks for several categories of objects, the target mask can be determined based on the selection operation entered by the user. The input mode includes, but is not limited to, a selection command such as a screen touch or a voice command. All master object masks that have an object category corresponding to an individual object selected by the user are target masks.

In particular, suppose that semantic segmentation is performed on the image to obtain k masks. These k masks correspond to different categories of objects. If k is greater than 2, the target mask is determined from the totality of these k masks in accordance with a selection command from the user. The area of the image corresponding to the target mask is defined as the target area, and the area of the image corresponding to the rest of the masks is defined as the background area.

As shown in FIG. 8, the set of masks for the output of the neural network contains masks A1, A2, B1 and B2 for the main objects and one background mask. Masks A1 and A2 belong to the same object category, and masks B1 and B2 also belong to the same object category. If the user presses, in the process of photographing, an individual object corresponding to mask A1 on the touch screen, masks A1 and A2 belonging to the same category of objects are determined as target masks, and the marking of masks B1 and B2 is changed and re-marked these masks as background masks. If the user presses, in the process of photographing, an individual object corresponding to mask B2 on the touch screen, masks B1 and B2 belonging to the same category of objects are determined as target masks, and the labeling of masks A1 and A2 is changed and re-labeled these masks as background masks.

It should be understood that the above specific embodiments of the invention are merely examples and should not be limiting. These specific embodiments of the invention described above can be freely combined without violating the logic of the invention. Therefore, after mask segmentation is performed on an image, multiple target masks can be obtained. These target masks may refer to one or more object categories, and each object category for these target masks may further span one or more individual objects. The result shown on the display refers to a rule that is set in the terminal system and that is for determining the target mask and for input from the user. In some scenarios, an image may alternatively contain only one background mask.

s224: Determination of the target area and the background area in the original image.

The original size of the photographed image is derived from the mask pool, with the target mask and the background mask from the mask pool also upsampled. The area composed by all pixels of the upsampled target mask and corresponding to the original image is the target area, and the area composed by all the pixels of the upsampled background mask and corresponding to the original image is the background area.

Step 23: Processing the target area and the background area in the image in different color processing modes to obtain the target image. The processing uses different color processing modes so that the color saturation in the target area becomes greater than the color saturation in the background area, or the brightness in the target area becomes greater than the brightness in the background area. In other words, the color saturation in the target area of the target image is greater than the color saturation in the background area, or the brightness in the target area of the target image is greater than the brightness in the background area of the target image.

In particular, the first color processing mode and the second color processing mode are used for the target area and the background area in the image, respectively. These first color processing mode and the second color processing mode include, but are not limited to, the following methods.

Method 1: The first color processing mode preserves the color, and the second color processing mode uses a filter, for example, converting the color of the background area to black and white. Typical filters include black and white filter, darkening filter, retro filter, film filter, blur filter, highlight filter (bokeh filter), and other similar filters.

For example, the black and white filter is designed to map each pixel value to a grayscale value in order to realize the black and white filter effect. As another example, the dimming filter is for reducing the brightness of each pixel value in order to achieve a special dimming effect.

Method 2: The first color processing mode is the first filtering method, the second color processing mode is the second filtering method, and the first filtering method is different from the second filtering method. For the same image, the color saturation of the image obtained using the first filtering method is greater than the color saturation of the image obtained using the second filtering method.

Method 3: The first color processing mode is the third filtering method, the second color processing mode is the fourth filtering method. For the same image, the color saturation of the image obtained using the third filtering method is greater than the color saturation of the image obtained using the fourth filtering method.

It should be clear that color is represented by both parameters - brightness and color saturation. Color saturation is a color property that does not cover brightness, but reflects the tone and saturation of the color itself, and brightness is the luminosity or degree of lightness of a color. Therefore, color processing comprises luminance processing and/or chroma processing.

In particular, filtering may include adjusting color saturation, brightness, and hue, and may further include texture mapping and other such factors. The color system can be purposefully adjusted by adjusting the color saturation and hue so that the color system becomes denser or lighter, or the hue changes while the other color system remains unchanged. The filtering process can also be understood as pixel-to-pixel mapping. The input image pixel value is mapped to the target pixel value using a predetermined mapping table to achieve a special effect. It should be understood that the filter may be a given parametric mask. Such color-related parameters may be filter mask parameters well known in the industry, or may be parameters independently assigned by the user.

In addition, after step 23, the method further comprises step 24 for saving the image processed in step 23.

According to the present invention, in the photographing process, the terminal can determine the target div and the background based on the content of the image, and perform different color processing on the target div and the background, so that the main object in the image photographed by the user can be more prominent, and the photographed image itself looks like film.

Example 2: Taking a photo of a video (movie recording mode)

In particular, according to the present invention, the image processing method for video recording is similar to the image processing method for taking photographs, and the difference is that the object processed in photographing is a single image, while the object processed in video recording is successive video frames. , namely several consecutive images. The object processed during video recording may be a full video or a segment of a full video, or a user-defined video clip within a certain period of time. For information about the processing procedure for each image in a video or video clip, refer to the processing method according to Example 1.

Specifically, the image processing method for photographing a video may include the following steps.

Step 31: Obtaining N photographed images, where N is a positive integer; and performing steps 32 and 33 on each picture, where the N pictures may be adjacent video frames such that the sum of the N pictures can be considered as video. Alternatively, these N pictures may not be adjacent.

One of the options for implementing step 32 may be the same as step 22.

One of the options for implementing step 33 may be the same as step 23.

In addition, since the video contains consecutive images, the method of determining the individual object also applies to the temporal sequence. Therefore, in addition to step 23, there may be more implementations in step 33. As an option, any method of determining the main object in step S223 may have a delay. For example, in the L1-th frame, a person and a background are defined, and they can still be defined in frames from the (L1 + 1)-th frame to the (L1 + L0)-th frame of the image by pixel marking and mask comparison, while the person in these images is the main subject, and the area corresponding to this person in the considered images is the target area 0. It is not necessary to define the main object and background for each frame. The timing at which the main object is determined may be set by the user each time, or the main object may be determined periodically, such as, but not limited to, every 2 s or every 10 s. The main object determination method in step S223 includes, but is not limited to, up to six methods.

Step 34: Saving a video composed of N images to which color processing has been applied.

According to the present invention, in the process of recording a video by a user, the terminal can determine the target div and the background based on the content of the video, and perform different color processing on the target div and the background, so that the main object in the video photographed by the user can be more highlighted. , the captured video itself can be as "cool" as a movie, and the user experience can be improved.

Example 3: Taking a photo of a video (movie recording mode)

According to the present invention, the image processing method for video recording is similar to the image processing method for photographing, and the difference is that the object processed in photographing is a single image, while the object processed in video recording is successive video frames, namely multiple consecutive images. Therefore, for information on the processing procedure for each image in a video or video clip, the processing method according to Example 1 can be referred to. In some complex video photographing scenarios, some areas of the image may not be received correctly. If the same area is separately marked in adjacent frames as the target area and background, this same area is processed in different colors in accordance with the above color processing method, and changing the colors of the same area in adjacent frames causes "touch flicker". Therefore, such flicker must be detected and eliminated during processing. This flickering can be understood as an error in determining the category of the object.

According to the video flicker detection method, the mask for the previous frame can be processed based on the optical flow to obtain the mask based on the optical flow, and the mask based on the optical flow can be compared with the mask of the current frame. When the degree of coincidence or degree of similarity exceeds a predetermined specific proportion, it is determined that no flicker occurs. When the degree of coincidence or degree of similarity does not exceed a predetermined specific proportion, it is determined that flicker has occurred. In addition, it should be understood that flicker detection is a continuous process. As an option, a specific method for determining if flicker is present is as follows:

(1) First, the optical flow of adjacent frames is calculated, where the optical flow serves as an indicator of the offset ratio between pixels in adjacent frames (in the (t - 1)th frame and in the tth frame).

(2) Obtain a mask for the (t - 1)th frame and calculate the mask F for the optical flow for the tth frame based on the mask for the (t - 1)th frame and the optical flow information for (t - 1)- the th frame and the t-th frame, where the mask for the optical flow is obtained through calculations based on the optical flow.

(3) Get the mask S for the t-th frame.

(4) The pixel set SF of the main object in the mask F for the optical flow is counted, and the pixel set SS of the main object in the mask S is counted. When (Nu - Ni)/Nu is greater than a predetermined threshold value, it is considered that there is a relatively large difference between the masks of neighboring frames of the (t - 1)th and tth, and it is determined that flicker occurs between (t - 1) - th frame and t-th frame, or it can be understood that the t-th frame flicker occurs. A relatively large difference means that the same object can be erroneously assigned to different categories of objects. For example, the same individual object in the (t - 1)-th frame and the t-th frame can be separately defined as a human and as a monkey.

As an option, if in the first N0 (positive integer greater than 2) images of the current shooting (current image), the number of groups of neighboring images with the same object defined in these images as belonging to different categories of objects is greater than the specified threshold value, it may be determined that it is necessary to perform a de-flickering procedure on the current frame. If it is determined that the number of neighboring picture groups in which the same object is determined to belong to different object categories does not exceed a predetermined threshold, it may be decided that it is not necessary to perform the flicker removal procedure on the current frame.

Optionally, for example, for a given number of historically adjacent frames or a given number of historical frames, if it is determined that flicker occurs in more than half of these frames, (for example, if it is determined that flicker occurs in three video frames out of the first five adjacent frames relative to current video frame) can be set to perform a flicker removal procedure on the current frame. If it is determined that the flicker occurs in less than half of these frames (for example, if it is determined that the flicker occurs in one of the first five adjacent frames relative to the current video frame), it can be determined that it is not necessary to perform the de-flicker procedure on the current frame. .

It should be clear that the current video image can be considered as an image recorded at some point in time. This moment can be understood as a generalized moment in some scenarios, or it can be understood as a specific moment in some scenarios, for example, the most recent moment or the moment that the user is interested in.

Specifically, the image processing method for photographing a video in this example may include the following steps.

Step 41: Obtaining N photographed images, where N is a positive integer; and carrying out the operations of steps 32 and 33 for each picture, where the N pictures can be adjacent video frames, and then the sum of these N pictures can be considered as video; or these N images may not be adjacent.

Step 42: Determining whether the number of adjacent picture groups in which flickering occurs in the first N0 frames relative to the current frame (current picture) is greater than a predetermined threshold. Here, the number N0 and the threshold value can be set by the user. For example, the number N0 represents a selected number of samples of historical video frames, and the threshold value may be 1/2, 2/3, or another similar fraction of the number N0. This is just an example and is not a limitation.

If the result of such a determination is not greater than a predetermined threshold value, the operations of steps 42 and 43 are performed with respect to the photographed or currently captured image.

One implementation of step 43 may be the same as step 32.

One implementation of step 44 may be the same as step 33.

If the determination result is greater than the predetermined threshold, the operation of step 44 is performed on the currently photographed or currently captured image.

Step 45: Processing all image areas in the current frame using the same color processing method to obtain the target image. This same color processing method may be the same as the color processing method of the background area in the previous frame, or may be the same as the color processing method of the target area in the previous frame, or may be the same as the color processing method of the entire images in the previous frame. For example, a color processing method that is the same as that used for the background area in step 33 (23) can be applied to the entire image. Alternatively, a color processing method that is the same as the method used for the target area in step 33(23) may be used for the entire image. For example, the entire image remains in color, or the entire image is black and white, or the first or second color processing mode (including but not limited to the color processing modes specified in Example 1) is used for the entire image.

In such a case, for the current frame, a mask segmentation procedure similar to that used in step 22 may be performed or omitted. In the present example, this is not limited in any way.

After step 45, step 46 is performed to save a video consisting of N images to which color processing has been applied. Here N is a positive integer.

According to the present invention, during a video recording procedure by a user, the terminal can determine the target div and background based on the content of that video, and perform different color processing on the target div and the background so that the main object in the video photographed by the user can be more highlighted, itself the photographed video could be as "cool" as the movie, and the user experience could be improved.

Example 4: Taking a picture of a video (movie recording mode)

In some application scenarios, the content of the image taken by the user is usually changed. Therefore, the main object of the image usually changes. The user also expects to be able to freely select the color processing mode of the main object in different images in order to independently control the style of the video.

The image processing method in the video photographing process may include the following steps.

Step 51: The terminal receives a video frame.

Step 52: The terminal determines the main object area and the background area in any video frame obtained from the video.

Step 53: The terminal uses any color processing mode at any time for the main object area, and this terminal uses any color processing mode at any time for the background area. However, you must be sure that for any image, the brightness or color saturation of the main object area after color processing will be greater than the brightness or color saturation of the background area after color processing. Alternatively, for any image, the color saturation or brightness of the image produced in the color processing mode used for the main subject area is greater than the color saturation or brightness of the image produced in the color processing mode used for the background region.

Example 5: Taking a photo of a video (movie recording mode)

In some application scenarios, the content of the image taken by the user is usually changed. Therefore, the main object of the image usually changes. The user also expects to be able to freely select the color processing mode of the main object in different images in order to independently control the style of the video. In particular, the color changes after a period of time.

The image processing method in the video photographing process may include the following steps.

Step 61: Capturing N1 images during the first time period and capturing N2 images during the second time period, where the first time period and the second time period are adjacent time periods, and both N1 and N2 are positive integers, these first the time period and the second time period may be of such duration that the user can identify image changes with the naked eye, and the numbers N1 and N2 are determined by frame rates and lengths of time periods during video recording. In this example, this is not limited in any way.

Step 62: Determining the first target area and the first background area in each of the N1 images, where the first background area is an image area different from the first target area, and the first target area in each of the N1 images corresponds to the first object (which may be at least one object) and defining a second target area and a second background area in each of the N2 images, where the second background area is an image area different from the second target area, and the second target area in each of the N2 images corresponds to a second object (which may be at least one object).

Step 63: Processing the first target region in the first color processing mode, processing the first background region in the second color processing mode, processing the second target region in the third color processing mode, and processing the second background region in the fourth color processing mode, to obtain a target video, where in this target video, the color saturation in the first target area is greater than the color saturation in the first background area, or the brightness of the first target area is greater than the brightness of the first background area; and the color saturation in the second target area is greater than the color saturation in the second background area, or the brightness of the second target area is greater than the brightness of the second background area.

Example 6: Taking a photo of a video (movie recording mode)

In some application scenarios, the content of the image taken by the user is usually changed. Therefore, the main object of the image usually changes. The user also expects to be able to freely select the target main object that the user expects to highlight in different images. For example, the image area corresponding to the first object is determined as the target area in the first time period, the image area corresponding to the second object is determined as the target area in the second time period, while the first object and the second object are different objects, different individual objects or belong to different categories of objects.

In this scenario, the image processing method in the video photographing process may include the following steps.

One implementation of step 71 may be the same as step 61.

Step 72: Determining a first target area and a first background area in any one of the N1 images based on the content of that image, and determining a second target area and a second background area in any one of the N2 images based on the content of that image, where the object or the object category corresponding to the second target area is different from the object or object category corresponding to the first target area, so that the system and the user can independently select the target main object and the target image area. The image contains the main object and the background, and accordingly, this image contains the target area and the background area.

For example, the first object is a human and the second object is an animal. For example, the first object is person A and the second object is person B. For example, the first object represents two people and the second object represents a dog and two cats. The rest of the area that has not been identified is marked as background.

In this method, the mask for the image may be determined using the approaches described above for steps S221 and S222. However, the following method is not limited to defining the target in the mask for each image.

As an option, in the image mask, the user can freely enter the first object and the second object, where these first object and the second object are determined in accordance with the selection command entered by the user. For example, if the user has selected an individual object, the system identifies the pixel corresponding to the command entered by the user, then identifies the particular (/some) individual object(s) (/there may be at least one individual object) or the specific (/some) category of the object. (/categories) (/can be at least one object category) for the mask selected by the user, further defines the specific (/some) individual object(s) or all individual objects that belong to the specific (/some) object category (/categories of objects) as the first object, and defines this first object or an area of the image corresponding to this first object as the first target area. This situation can be maintained for some period of time. In other words, in the next few frames, the area corresponding to the mask corresponding to the first object is the first target area until the user selects another individual object at the next time, and the area corresponding to the new individual object is determined as the second target area. in accordance with a method similar to the method described above. In any image, an area of the image other than the first target area or the second target area is the background area. More specifically, the area corresponding to the mask corresponding to the first object during the first time period is the first target area, and the area corresponding to the mask corresponding to the second object during the second time period is the second target area.

Optionally, in the image mask pool, the system may determine the target image mask over a period of time in the image mask pool based on a given amount of time (for example, but not limited to 1 s or 2 s) or a given number of frames. (for example, but not limited to 50 frames or 100 frames). For example, the first target mask is determined in the 101st frame, and a mask having the same object category or individual object as the first target mask is used as the first target mask for each of the subsequent frames from frame 102 to frame 200 until until the second target mask is determined in the 201st frame. For each of the subsequent frames from 202 to 200 frames, a mask having the same object category or the same individual object as the specified second target mask is used as the second target mask. It should be understood that the numbers in the above example may be predefined by the user or the system.

For information on the method for determining the first target mask and the second target mask, one can refer to, but not limited to, any of the six methods described in step S223. Therefore, the first target mask and the second target mask may refer to the same object category or the same individual object, or they may refer to different object categories or different individual objects. This refers to the ability to identify a communication network, change the image of a scene, or a user-entered command.

More specifically, a target mask is determined at some point in time, and then a mask of that type, or a mask for that individual object, is continuously used for a period of time.

In addition, the first target area, the first background area, the second target area, and the second background area are then determined according to the method described in step s224.

One implementation of step 73 may be the same as step 63.

In addition, since this example may vary depending on the time period, several combinations of color processing methods are possible.

For example, the first color processing mode is the same as the third color processing mode, and the second color processing mode is the same as the fourth color processing mode. This color processing mode has good consistency.

For example, the first color processing mode is the same as the third color processing mode, and the second color processing mode is different from the fourth color processing mode. In this example of color processing modes, the colors of the target main object are matched, and the background colors are changed so that the overall visual effect is more brilliant.

For example, the first color processing mode is different from the third color processing mode, and the second color processing mode is the same as the fourth color processing mode. In this example of color rendering modes, the background colors are matched, and the colors of the target main object are changed so that the target main object is more prominent.

For example, the first color processing mode is different from the third color processing mode, and the second color processing mode is different from the fourth color processing mode. In this example of color processing modes, more color conversion methods can be implemented, and more color cooperation can be implemented based on the requirements of different scenarios.

The first color processing mode or the third color processing mode contains a filter, for example, to preserve color or enhance color. The second color processing mode or the fourth color processing mode contains filters such as a black and white filter, a darkening filter, a retro filter, a film filter, a blur filter, and a flare filter (bokeh filter).

More specifically, for information on the color processing methods for the target area and the background area of the same image, refer to step 23. For these N2 images, the third color processing mode and the fourth color processing mode are respectively the same as the first color processing mode and the second color processing mode. .

According to the above technical solutions, in some examples, the user can freely select the background color processing mode in different images to highlight different backgrounds. In some scenarios, the user can freely select the color mode of the main object in different images to emphasize, highlight the main object in different degrees or forms.

It should be understood that in different examples of the present invention, signals designated by the same label may have different sources or may be obtained in different ways. This is not a limitation. In addition, when referring to steps from various examples, "same as step xx" is more focused on the fact that the signal processing logic of the two steps are similar. It is not limited to the fact that the input and output signals of the two types must be exactly the same and that the procedures of these two methods are completely equivalent. Appropriate references and variations, which may be made by a person skilled in the art, fall within the protection scope of the present invention.

The present invention provides an image processing method. In an image, a target area and a background area are determined by performing mask segmentation for that image. Different color processing modes are applied to the target area and the background area, so that the brightness of the target area becomes larger than the brightness of the background area, or the color saturation of the target area becomes higher than the color saturation of the background area, so that the main object is more highlighted. This realizes the special effect of cinema.

Based on the image processing method discussed in the above embodiment, one embodiment of the present invention provides an image processing apparatus 900. This apparatus can be used in multiple terminals and can be applied to any form of implementation of the terminal 100, such as a terminal having a video capture function. As shown in FIG. 9, the instrument contains:

photographic module 901 configured to acquire an image that may be the result of photographing or photographing a video, where this module is specifically configured to carry out the method according to step 21, step 31, step 51, step 61 or step 71 of the above example or any or a method equivalent to this method, and this module can be implemented by the processor by calling the appropriate program instructions from the storage device to control the video camera to capture an image;

a decision module 902 configured to determine a target area and a background area in an image based on the content of that image, where the decision module is specifically configured to perform the method according to step 22, step 32, step 52, step 62, or step 72 of the above example, or whichever - or a method equivalent to this method, and this module can be implemented by the processor by calling the appropriate program instructions from the storage device to execute the appropriate algorithm; And

a color processing unit 903 configured to use different color processing modes for the target area and the background area in the image to obtain the target image or the target video, so that the color saturation in the target area becomes greater than the color saturation in the background area, or the brightness in the target area becomes above the brightness in the background area, where this module is specifically configured to implement the method according to step 23, step 33, step 53, step 63 or step 73 of the above example or any method equivalent to this method, and this module can be implemented through the processor by calling the appropriate program instructions from a storage device using a special algorithm.

In addition, the apparatus may further comprise a data storage module 904 configured to store an image or video that has been color processed.

The above specific examples of the method, explanations and descriptions of the technical features of variants and extensions of several forms of implementation also apply to the implementation of this method in the considered equipment, so details in relation to this equipment will not be described.

The present invention provides an image processing apparatus. With regard to the image, segmentation of the mask is performed to determine the target area and the background area of the image based on the content of this image. Different color processing modes are applied to the target area and the background area, so that the brightness of the target area becomes higher than the brightness of the background area, or the color saturation in the target area becomes greater than the color saturation in the background area, so that the main object corresponding to the target area is more highlighted. This realizes the special effect of cinema.

Based on the image processing method discussed in the above embodiment, one embodiment of the present invention provides an image processing apparatus 1000. This apparatus can be used in multiple terminals and can be applied to any form of implementation of the terminal 100, such as a terminal having a video capture function. As shown in FIG. 10, the instrument contains:

photographic module 1001 configured to acquire an image that may be the result of photographing or photographing a video, where this module is specifically configured to perform the method according to step 21, step 31, step 51, step 61 or step 71 of the above example or any or a method equivalent to this method, and this module can be implemented by the processor by calling the appropriate program instructions from the storage device to control the video camera to capture an image;

an evaluation module 1002, configured to: determine whether the number of flickering frames in the set of first N0 frames relative to the current frame is greater than a predetermined threshold; and if, according to the determination result, the number of flickering frames is not more than a predetermined threshold, continue by starting the decision module 1003 and the color processing module 1004 to perform the respective functions; or if, according to the determination result, the number of flickering frames is greater than a predetermined threshold, continue by starting the flicker elimination module 1005 to perform the corresponding function; and the module 1002 is specifically configured to carry out the method according to step 42 in the above example and any method that may be equivalent to said method, and this module may be implemented by the processor by calling appropriate program instructions from memory to execute the corresponding algorithm ;

a decision module 1003, configured to: when the evaluation module 1002 determines that the number of flickering frames from the set of first N0 frames relative to the current frame is not greater than a predetermined threshold, determine the target area and the background area in the image based on the content of this image, where this the module is specifically configured to perform the method according to step 22, step 32, step 43, step 52, step 62, or step 72 of the above example, or any method equivalent to this method, and this module can be implemented by the processor by calling the appropriate program commands from the storage device to perform the corresponding algorithm; And

a color processing unit 1004, configured to use different color processing modes for the target area and for the background area in the image, so that the color saturation in the target area becomes greater than the color saturation in the background region, or the brightness in the target region becomes higher than the brightness in the background region, where this module is specifically configured to implement the method according to step 23, step 33, step 44, step 53, step 63 or step 73 of the above example or any method equivalent to this method, and this module can be implemented by the processor by calling corresponding program commands from a storage device using a special algorithm; And

a flicker elimination module 1005, configured to: when the evaluation module 1002 determines that the number of flickering frames in the aggregate of the first N0 frames relative to the current frame is greater than a predetermined threshold, to use the same color processing method for all image areas in the current frame , where this same color processing method may be the same as the color processing method of the background area in the previous frame, or may be the same as the color processing method of the target area in the previous frame, and the module in question is specially configured to implement the method according to step 45 of the above example, or some method equivalent to this method, and this module can be implemented by the processor by calling the appropriate program instructions from a memory device using a special algorithm.

In addition, apparatus 100 may further comprise a data storage module 1006 configured to store an image or video that has been color processed.

The above specific examples of the method, explanations and descriptions of the technical features of variants and extensions of several forms of implementation also apply to the implementation of this method in the considered equipment, so details in relation to this equipment will not be described.

The present invention provides an image processing apparatus. With regard to the image, segmentation of the mask is performed to determine the target area and the background area of the image based on the content of this image. Different color processing modes are applied to the target area and the background area, so that the brightness of the target area becomes higher than the brightness of the background area, or the color saturation in the target area becomes greater than the color saturation in the background area, so that the main object corresponding to the target area is more highlighted. This realizes the special effect of cinema.

It should be understood that the division of the equipment discussed above into modules is only a division of logical functions. In an actual implementation, some or all of the modules may be integrated in a single physical entity, or may be physically separate. For example, each of the modules mentioned above may be a separate processing element, or it may be integrated on an integrated circuit chip in the terminal hardware, or may be stored in a memory element of the controller in the form of program code. The processing element of the processor calls and executes the function of each of the modules mentioned above. In addition, the listed modules may be integrated or may be implemented independently. Here, the processing element may be an integrated circuit chip and have signal processing capability. During implementation, the steps of the above methods or the above modules may be implemented using hardware integrated logic circuits in the processing element, or using instructions in the form of software. The processing element may be a general purpose processor, such as a central processing unit (CPU), or may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits. circuits (English: application-specific integrated circuit, abbreviated ASIC), or one or more microprocessors or digital signal processors (English: digital signal processor, abbreviated DSP), or one or more field-programmable gate arrays (English: field-programmable gate array , FPGA for short).

It should be understood that in the description, the Claims and the accompanying drawings of the present invention, the terms "first", "second" and other similar terms are intended to distinguish between like objects, but do not necessarily indicate a specific order or sequence. It should be understood that data so designated are interchangeable in appropriate circumstances, so that the options described here may be implemented in a different order than the order illustrated or described here. In addition, the terms "includes", "comprises" and any other variations are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device having some list of steps or modules, not necessarily are limited only to the steps or modules explicitly listed in this list, and may also contain other steps or modules that have not been explicitly listed or that are inherent in such a process, method, product or device.

One skilled in the art will understand that embodiments of the present invention may be implemented as a method, system, or computer program product. Therefore, the present invention may take the form of purely hardware variants, purely software variants, or variants that are a combination of hardware and software. Moreover, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) containing computer-usable program code.

The present invention has been described with reference to logic diagrams and/or flowcharts of a method, apparatus (system), and computer program product according to embodiments of the present invention. It should be understood that computer program instructions may be used to implement each process and/or each block in logic diagrams and/or block diagrams, and the combination of the process and/or block in logic diagrams and/or block diagrams. These computer program instructions may be intended for a general purpose computer, a special purpose computer, an embedded processor, or a processor of another programmable data processing device to generate a machine such that these instructions, when executed by a computer or processor of another programmable data processing device, generate hardware to perform a particular function in one or more processes from the logic diagrams and/or in one or more blocks from the block diagrams.

These computer program instructions may be stored in a computer readable storage device and may instruct a computer or other programmable processing device to operate in a particular manner such that the instructions stored in the computer readable storage device generate an artifact containing the command hardware. This command hardware implements a specific function in one or more of the processes from the logic diagrams and/or in one or more blocks from the block diagrams.

These computer program instructions may alternatively be loaded into a computer or other programmable data processing device such that the computer or programmable data processing device performs a series of operations and steps, thereby generating computer-implemented processing. Therefore, instructions executed by a computer or other programmable data processing device form steps for implementing a particular function in one or more processes of the logic circuits and/or in one or more blocks in the logic circuits.

While some embodiments of the present invention have been described herein, various changes and modifications may be made to those variations by those skilled in the art after learning the basic concepts of the invention. Therefore, the appended claims are to be construed to cover the listed variations and that all changes and modifications fall within the scope of the present invention. Obviously, a person skilled in the art can make various modifications and variations in the embodiments of the present invention, without deviating from the meaning and scope of these embodiments of the invention. The present invention is intended to cover these modifications and variations on the assumption that they fall within the protection scope of the appended claims and equivalent technologies.

Claims (153)

1. An image processing method, wherein the method comprises: capturing the first video when the terminal is in the video recording mode; And, in the process of capturing the first video, identifying the main object area and the background area in the image of the first video based on the category of the object using the neural network, where this area of the main object contains a person, animal, vegetation, vehicle, clothing or other specified object, and the background area is an image area in the first video other than the main subject area; And, for displaying the first video, keeping the color of the main object area and performing black and white processing of the background area to obtain the second video, in which the identification of the area of the main object and the background area in the image of the first video based on the category of the object contains one of the following methods: method 1: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k = 2 and the combination of the two masks contains one mask for the main object and one background mask, determining the area of the image corresponding to the mask for the main object as the main object area, and determining the area corresponding to the background mask as the background area; method 2: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, the image area corresponding to these k0 masks for the main objects is determined as the main object area, and the image area corresponding to one of the other masks, as the background region, where k0 is a non-negative integer less than k; method 3: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects, and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2, determining, as the area of the main object, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determining the area of the image corresponding to the rest of the masks, as the background area; method 4: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k is greater than 2, determine the mask for the main object from the set of k masks based on the given object category priorities, and determining an image area corresponding to a mask for the main object as the main object area, and determining an image area corresponding to the rest of the masks as a background area; or method 5: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k is greater than 2, determining a mask for the main object from the set of k masks in accordance with the selection command from the user, and determining an image area corresponding to the mask for the main object as the main object area, and determining an image area corresponding to the rest of the masks as the background area. 2. The method according to claim 1, in which, after receiving the second video, this method further comprises: capturing a third video, where the content of this third video is different from the content of the first video; in the process of capturing the third video, if the neural network is used to identify only the background in the image of the third video based on the category of the object, performing black and white image processing of the third video to obtain a fourth video. 3. The method according to claim 1 or 2, in which the neural network is able to identify at least two categories of objects from the list - a person, an animal, vegetation, a vehicle (car), clothing, or another specified category of an object. 4. The method according to any one of paragraphs. 1 to 3, in which the area of the main object is obtained by performing image segmentation at the pixel level using a neural network. 5. The method according to any one of paragraphs. 1-4, in which the neural network is obtained by tuning using a set of tuning data, where the set of tuning data contains input images and mask graphs for several application scenarios. 6. The method according to any one of paragraphs. 1 to 5, in which the areas of the main objects for all images of the second video correspond to the same object. 7. The method according to any one of paragraphs. 1 - 5, in which the main object area contains one or more objects of the same category; or the main object area contains objects of several different categories. 8. The method according to any one of paragraphs. 1 to 7, wherein the method further comprises performing a background area filtering processing, where the filtering processing comprises at least one of a highlight filtering (bokeh processing), a darkening processing, a retro processing, or a texture mapping procedure. 9. Equipment for image processing, in which this equipment contains: a photographic module configured to capture a first video when the terminal is in a video recording mode; a solver configured to, in the process of capturing the first video, identify the main object area and the background area in the first video image based on the category of the object using a neural network, where the main object area contains a person, animal, vegetation, vehicle, clothing, or object another predetermined category, and the background area is the image area of the first video, different from the area of the main object; And a color processing module configured to preserve the color of the main object area for the first video image and perform black and white processing of the background area to obtain the second video, wherein the solver is specifically configured to: perform semantic segmentation of the image to obtain k masks, where these k masks correspond to different categories of objects and the totality of these k masks contains a background mask and masks for the main objects; And, if k = 2 and the combination of the two masks contains one mask for the main object and one background mask, determining the area of the image corresponding to the mask for the main object as the main object area, and determining the area corresponding to the background mask as the background area; if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, the image area corresponding to these k0 masks for the main objects is determined as the main object area, and the image area corresponding to one of the other masks, as the background region, where k0 is a non-negative integer less than k; if k is greater than 2, determining, as the area of the main object, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determining the area of the image corresponding to the rest of the masks, as the background area; if k is greater than 2, determine the mask for the main object from the set of k masks based on the given object category priorities, and determine the image area corresponding to the mask for the main object as the main object area, and determine the image area corresponding to the rest of the masks as the background areas; or, if k is greater than 2, determining a mask for the main object from the set of k masks in accordance with a selection command from the user; And determining an image area corresponding to the mask for the main object as the main object area, and determining an image area corresponding to the rest of the masks as the background area. 10. The apparatus of claim 9, wherein the photographic module is further configured to capture a third video; the content of this third video is different from the content of the first video; and in the process of capturing the third video, the color processing module is further configured to perform black and white image processing of the third video to obtain a fourth video when the decider identifies only the background in the third video image based on the object category using the neural network. 11. Equipment according to claim 9 or 10, in which the neural network is able to identify at least two categories of objects from the list - a person, an animal, vegetation, a vehicle (car), clothing, or another specified category of an object. 12. Equipment according to any one of paragraphs. 9 to 11, in which the area of the main object is obtained by performing image segmentation at the pixel level using a neural network. 13. Equipment according to any one of paragraphs. 9 to 12, in which the neural network is obtained by tuning using a set of tuning data, where the set of tuning data contains input images and mask graphs for several application scenarios. 14. Equipment according to any one of paragraphs. 9 to 13, in which the main object areas for all images of the second video correspond to the same object. 15. Equipment according to any one of paragraphs. 9 to 14, in which the main object area contains one or more objects of the same category; or the main object area contains objects of several different categories. 16. Equipment according to any one of paragraphs. 9 to 15, in which the color processing module is further configured to perform background area filter processing, where the filter processing comprises at least one of highlight filtering (bokeh processing), darkening processing, retro processing, or texture mapping procedure. 17. Terminal for image processing, in which this terminal contains a video camera, a storage device, a processor and a bus; so that the video camera, storage device, and processor are connected via a bus; the video camera is configured to capture an image; the storage device is configured to store a computer program and instructions; And the processor is configured to call the computer program, instructions, and the captured image stored in the storage device to implement the method according to any one of paragraphs. 18. 18. The terminal of claim 17, wherein said terminal further comprises an antenna system, and said antenna system receives/transmits a radio communication signal under control of the processor for radio communication with a mobile communication network, wherein the mobile communication network comprises one or more of the following communication networks: GSM communication, CDMA communication network, 5G communication network, FDMA communication network, TDMA communication network, PDC communication network, TACS communication network, AMPS communication network, WCDMA communication, TDSCDMA communication network, Wi-Fi communication network and LTE communication network. 19. An image processing method, wherein the method comprises: obtaining N images, where N is a positive integer and these N images constitute a video; And performing step 1 and step 2 on each of these N images to get the target video: step 1: determining the target area and the background area in the image based on the category of the object in this image, where the background area is an area of the image other than the target area; And stage 2: processing the target area and the background area in different color processing modes, in which the procedure for determining the target area and the background area in the image based on the category of the object in this image contains one of the following methods: method 1: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area; method 2: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, determine the image area corresponding to these k0 masks for the main objects as the target area, and determine the image area corresponding to one from the remaining masks, as a background region, where k0 is a non-negative integer less than k; method 3: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects, and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2, determine, as the target area, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determine the area of the image corresponding to the rest of the masks, as the background area; method 4: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k is greater than 2, determine the target mask from the set of k masks based on the given object category priorities, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the rest of the masks as a background area; or method 5: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k is greater than 2, determining the target mask from the set of k masks according to the selection command from the user, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the remaining masks as a background area. 20. The method of claim 19, wherein the procedure for processing the target area and the background area in different color processing modes comprises: preserving the color of the target area and performing black and white processing of the background area. 21. The method of claim 19, wherein the procedure for processing the target area and the background area in different color processing modes comprises: preserving the color of the target area; and performing dimming processing of the background area. 22. The method of claim 19, wherein the procedure for processing the target area and the background area in different color processing modes comprises: preserving the color of the target area and performing "blurring" processing of the background area. 23. The method according to any one of paragraphs. 19 - 22, in which the set of categories of objects contains at least one of the list - a person, animal, vegetation, a given object or background. 24. An image processing method, wherein the method comprises: getting an image; determining a target area and a background area in the image based on the category of the object in the image, where the background area is an area of the image other than the target area; And preserving the color of the target area and performing "blurring" processing of the background area to obtain the target image, wherein defining the target area and background area in an image based on the category of the object in that image contains one of the following: method 1: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area; method 2: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, determine the image area corresponding to these k0 masks for the main objects as the target area, and determine the image area corresponding to one from the remaining masks, as a background region, where k0 is a non-negative integer less than k; method 3: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects, and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2, determine, as the target area, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determine the area of the image corresponding to the rest of the masks, as the background area; method 4: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k is greater than 2, determine the target mask from the set of k masks based on the given object category priorities, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the rest of the masks as a background area; or method 5: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k is greater than 2, determining the target mask from the set of k masks according to the selection command from the user, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the remaining masks as a background area. 25. An image processing method, wherein the method comprises: obtaining N images, where N is a positive integer and these N images constitute a video; determining the category of the object included in each received image; And, if in the first N0 images of the current shooting (current image), the number of groups of adjacent images with the same object, defined as different categories of objects, is not more than a specified threshold, determining the target area and background area in the current image based on the category of the specified object in the image where the background region is an area of the current image that is different from the target region; And processing the target area and the background area in the first color processing mode and the second color processing mode, respectively, to obtain a target image, where the first color processing mode is different from the second color processing mode; or, if in the first N0 images of the current shooting (current image) the number of groups of neighboring images with the same object, defined as different categories of objects, is greater than a predetermined threshold, processing this current image in the first color processing mode or in the second color processing mode to obtain target image, where this current image is one of the specified N images. 26. The method of claim 25, wherein the first color processing mode comprises color preservation and the second color processing mode comprises black and white processing, dimming processing, or "blurring" processing. 27. The method according to claim 25 or 26, in which the set of categories of objects contains at least one category from the list - a person, animal, vegetation, a given object or background. 28. The method according to any one of paragraphs. 25 to 27, in which the procedure for determining the target area and the background area in the current image based on the category of the object in this image contains one of the following methods: method 1: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area; method 2: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, determine the image area corresponding to these k0 masks for the main objects as the target area, and determine the image area corresponding to one from the remaining masks, as a background region, where k0 is a non-negative integer less than k; method 3: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects, and the set of these k masks contains a background mask and masks for the main objects; And, if k is greater than 2, determine, as the target area, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determine the area of the image corresponding to the rest of the masks, as the background area; method 4: performing semantic image segmentation to obtain k masks, where these k masks correspond to different categories of objects; And, if k is greater than 2, determine the target mask from the set of k masks based on the given object category priorities, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the remaining masks as a background area. 29. Equipment for image processing, in which this equipment contains: a photographic module configured to take N images, where N is a positive integer and the N images constitute a video; a decision module configured to determine the target area and the background area in each of the specified N images based on the category of the object in this image, where the background area is an image area different from the target area; And a color processing module configured to process the target area and the background area in different color processing modes, wherein the solver is specifically configured to: perform semantic segmentation of the image to obtain k masks, where these k masks correspond to different categories of objects and the totality of these k masks contains a background mask and masks for the main objects; And, if k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area; if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, determine the image area corresponding to these k0 masks for the main objects as the target area, and determine the image area corresponding to one from the remaining masks, as a background region, where k0 is a non-negative integer less than k; if k is greater than 2, determine, as the target area, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determine the area of the image corresponding to the rest of the masks, as the background area; if k is greater than 2, determining a target mask from a plurality of k masks based on predetermined object category priorities, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the rest of the masks as a background area; or, if k is greater than 2, determining a target mask from the set of k masks in accordance with a selection command from the user; And determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the remaining masks as a background area. 30. The apparatus of claim 29, wherein the color processing module is specifically configured to: preserve color in the target area and perform black and white processing, dimming processing, or "blurring" processing of the background region. 31. Equipment for image processing, in which this equipment contains: a photographic module configured to take N images, where N is a positive integer and the N images constitute a video; a decision module configured to determine the category of the object included in each received image; And an evaluation module configured to: if, in the first N0 images of the current image, the number of groups of adjacent images with the same object, defined as different categories of objects, is not greater than a predetermined threshold, control the decision module to determine the target area and the background area in the current image based on the category of the specified object in the image, where the background area is an area of the current image other than the target area; and controlling the color processing unit to process the target area and the background area in the first color processing mode and the second color processing mode, respectively, to obtain a target image, where the first color processing mode is different from the second color processing mode; or, if in the first N0 images of the current image, the number of groups of neighboring images with the same object, defined as different categories of objects, is greater than a predetermined threshold value, control the flicker elimination module to process the current image in the first color processing mode or in the second color processing mode for obtaining the target image, where this current image is one of the specified N images. 32. The apparatus of claim 31, wherein said decision module is, in particular, further configured to: perform semantic segmentation of the image to obtain k masks, where these k masks correspond to different categories of objects and the totality of these k masks contains a background mask and masks for the main objects; And, if k = 2 and the combination of these two masks contains one mask for the main object and one background mask, determine the area of the image corresponding to the mask for the main object as the target area, and determine the area corresponding to the background mask as the background area; if k is greater than 2 and the number of pixels included in the k0 masks for the main objects from the set of the specified k masks is greater than a predetermined threshold value, determine the image area corresponding to these k0 masks for the main objects as the target area, and determine the image area corresponding to one from the remaining masks, as a background region, where k0 is a non-negative integer less than k; if k is greater than 2, determine, as the target area, the area of the image corresponding to the mask containing the largest number of pixels and included in the specified set of k masks, and determine the area of the image corresponding to the rest of the masks, as the background area; if k is greater than 2, determining a target mask from a plurality of k masks based on predetermined object category priorities, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the rest of the masks as a background area; or, if k is greater than 2, determining the target mask from the set of k masks according to the selection command from the user, and determining an image area corresponding to the target mask as a target area, and determining an image area corresponding to the remaining masks as a background area. 33. Apparatus according to claim 31 or 32, wherein the color processing module is specifically configured to: preserve the colors of the target area and perform black and white processing, dimming processing, or "blurring" processing of the background area. 34. Terminal for image processing, in which this terminal contains a video camera, a storage device, a processor and a bus; so that the video camera, storage device, and processor are connected via a bus; the video camera is configured to capture an image; the storage device is configured to store a computer program and instructions; And the processor is configured to call the computer program, instructions, and the captured image stored in the storage device to implement the method according to any one of paragraphs. 19 - 23. 35. The terminal of claim 34, wherein said terminal further comprises an antenna system, and said antenna system receives/transmits a radio communication signal under the control of the processor for radio communication with a mobile communication network, wherein the mobile communication network comprises one or more of the following communication networks: GSM communication, CDMA communication network, 3G communication network, 4G communication network, 5G communication network, FDMA communication network, TDMA communication network, PDC communication network, TACS communication network, AMPS communication network, WCDMA communication, TDSCDMA communication network, Wi-Fi network communication or LTE communication network. 36. Terminal for image processing, in which this terminal contains a video camera, a storage device, a processor and a bus; so that the video camera, storage device, and processor are connected via a bus; the video camera is configured to capture an image; the storage device is configured to store a computer program and instructions; And the processor is configured to call the computer program, instructions, and the captured image stored in the storage device to implement the method according to any one of paragraphs. 24 - 28. 37. The terminal of 36, wherein said terminal further comprises an antenna system, and said antenna system receives/transmits a radio communication signal under the control of a processor for radio communication with a mobile communication network, wherein the mobile communication network comprises one or more of the following communication networks: a GSM communication network , CDMA communication network, 3G communication network, 4G communication network, 5G communication network, FDMA communication network, TDMA communication network, PDC communication network, TACS communication network, AMPS communication network, WCDMA communication, TDSCDMA communication network, Wi-Fi communication network and LTE communication network.

Images