RU2794062C2 - Image processing device and method and equipment - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: technical result is to increase the accuracy of determining the target area and the background area. To achieve a technical result in the claimed solution, the area of the main object and the background area in the image are determined based on the category of the object in the image and the neural network; preserving the color of the main object area in the image and performing black and white color processing or blur processing in the background area in the image, determining the first mask of the first category of objects and the second mask of the second category of objects based on the neural network, the first mask defining the area of the main object, and the second mask defines the background area, and the first category of the object contains an object from a given category of objects; and the second category of the object is the background.

EFFECT: increase in the accuracy of determining the target area and the background area.

18 cl, 10 dwg

Description

The field of technology to which the invention belongs

The present description relates to the field of terminal technologies and, in particular, to an image processing apparatus and method and equipment.

State of the art

Photographing involves using a camera or video recorder to record an image of a person or object. Currently, different photography skills are used for different scenes, such as night scene photography, rain scene photography, building photography, and portrait photography. Dynamic photography for cinema is also a type of photography, but must follow a specific principle. With the development of science and technology, photography has become easier and more popular.

With the increase in network bandwidth and the increase in the processing capability of end devices, video and photo taking and video and image sharing become more convenient, and video use has become a new daily need for all people. Video is rapidly becoming a heavy service in network traffic and is expected to account for 80% to 90% of total traffic in the coming years.

In everyday life, photography has become the main way to illustrate oneself for users and enjoy the beauty of things. People want to take pictures with more remarkable styles. For example, using a special image processing or video effect during photography to convey the impression of photography "What you see is what you get." Therefore, for non-professional users of the terminal device, it is necessary to introduce new image processing technologies.

At present, the video recording function of the terminal device is monotonous. At present, only the ordinary way of photographing can be provided, which does not have some personalized effects.

Disclosure of the essence of the invention

The present description proposes an image processing method. In the image, the target area and the background area are determined by performing segmentation of the mask (pattern) on the image. Different color processing modes are applied to the target area and the background area, so that the luminance of the target area is higher than the luminance of the background area, or the chroma of the target area is larger than the chroma of the background area, and the main object corresponding to the target area is more prominently highlighted. This allows the user of the terminal device to get the effect of a video clip while taking a photograph or shooting a video, and improves the transmission of the user's photographing experience.

Specific technical solutions provided in the embodiments of the present description are as follows.

According to a first aspect, an embodiment of the present disclosure provides an image processing method. The method is applied to a video recording process, and the method includes: capturing N1 images in a first time period; capturing N2 images in a second time period, where the first time period and the second time period are adjacent time periods, and N1 and N2 are positive integers; determining a first target area and a first background area from each of the N1 images, where the first background area is an image area different from the first target area, and the first target area in each of the N1 images corresponds to the first object; determining a second target area and a second background area from each of the N2 images, where the second background area is an image area different from the second target area, and the second target area in each of the N2 images corresponds to the second object; and processing the first target region in the first color processing mode, processing the first background region in the second color processing mode, processing the second target region in the third processing mode, and processing the second background region in the fourth color processing mode to obtain the target video. In the target video, the chroma of the first target area is higher than the chroma of the first background area, or the luminance of the first target area is higher than the luminance of the first background area. Further, the chroma of the second target area is greater than the chroma of the second background region, or the luminance of the second target region is greater than the luminance of the second background region.

According to a second aspect, an embodiment of the present disclosure provides an image processing apparatus. The device is used in a video capturing process, and the device includes: a photographing unit configured to capture N1 images in a first time period and capture N2 images in a second time period, where the first time period and the second time period are adjacent time periods, and N1 so N2 are positive integers; a determination module configured to: determine a first target area and a first background area from each N1 images, where the first background area is an image area different from the first target area and the first target area in each N1 image corresponds to the first object; and determining a second target area and a second background area from each N2 images, where the second background area is an image area different from the second target area and the second target area in each N2 image corresponds to the second object; and a color processing unit configured to process the first target area in the first color processing mode, process the first background area in the second color processing mode, process the second target area in the third color processing mode, and process the second background area in the fourth color processing mode to obtain the target video. In the target video, the chroma of the first target area is greater than the chroma of the first background area, or the luminance of the first target area is higher than the luminance of the first background area. Further, the chroma of the second target area is greater than the chroma of the second background area, or the luminance of the second target area is higher than the luminance of the second background area.

According to the first aspect or the second aspect, in a possible implementation, the first object and the second object are the same object.

According to the first aspect or the second aspect, in a possible implementation, the first object and the second object are different objects.

According to the first aspect or the second aspect, in a possible implementation, the first object or the second object includes at least one single item of the object category (eg, human, animal or plant).

According to the first aspect or the second aspect, in an exemplary implementation, the first object and the second object are determined in accordance with the user's selection instruction.

Specifically, for example, the first object is determined from the first image in the first time period according to the user selection instruction, and the first object is used as the target object in all images in the first time period. Similarly, the second object is determined from the first image in the second time period according to the user selection instruction, and the second object is used as the target object in all images in the second time period. For example, semantic segmentation can be performed to obtain k masks on an image. The k masks correspond to different categories of the object, the selection instruction entered by the user corresponds to the specific mask or specific masks, and the object corresponding to the specific mask or specific masks is the target object.

According to a first aspect or a second aspect, in an exemplary implementation, the first object and the second object are separately determined by the terminal device based on the content of the two pictures in a given time interval.

Specifically, for example, the first object is determined from the first image in the first time period, and the first object is used as the target object in all images in the first time period. Similarly, the second object is determined from the first image in the second time period, and the second object is used as the target object in all images in the second time period. The method in which the first object is determined from the first image in the first time period and the second object is determined from the first image in the second time period includes, but is not limited to, one of the following methods:

performing segmentation of semantics on the image to obtain k masks, where k masks correspond to different categories of the object; And

if k=2 and the two masks include one object mask and one background mask, specifying an image area corresponding to the object mask as the target area, and determining the area corresponding to the background mask as the background area, where the object corresponding to the mask object is the first object or the second object;

if k exceeds 2 and the number of pixels included in the k0 object masks in the k masks exceeds a predetermined threshold value, determining the image area corresponding to the k0 object masks as the target area, and determining the image area corresponding to the remaining masks as the background area where the object , corresponding to the object mask, is the first object or the second object, and k0 is a non-negative integer less than k;

if k is greater than 2, determining, as the target area, the area of the image corresponding to the mask that includes the largest number of pixels, that is, in k masks, and determining the area of the image corresponding to the remaining masks as the background area where the object, corresponding to the mask of the object, is the first object or the second object;

if k is greater than 2, determining a target mask of the k masks based on predetermined object category priorities; and defining an image area corresponding to the target mask as a target area, and defining an image area corresponding to the remaining masks as a background area, where the object corresponding to the object mask is the first object or the second object; or

if k is greater than 2, determining a target mask of the k masks according to the user selection instruction; and determining an image area corresponding to the target mask as the target area, and determining an image area corresponding to the remaining masks as the background area, where the object corresponding to the object mask is the first object or the second object. The method may be specifically performed by the determination module.

According to the first aspect or the second aspect, in an exemplary implementation, the first color processing mode is the same as the third color processing mode, and the second color processing mode is the same as the fourth color processing mode.

According to the first aspect or the second aspect, in an exemplary implementation, the first color processing mode is the same as the third color processing mode, and the second color processing mode is different from the first color processing mode.

According to the first aspect or the second aspect, in an exemplary implementation, the first color processing mode is different from the third color processing mode, and the second color processing mode is the same as the fourth color processing mode.

According to the first aspect or the second aspect, in an exemplary implementation, the first color processing mode is different from the third color processing mode, and the second color processing mode is different from the fourth color processing mode.

According to the first aspect or the second aspect, in an exemplary implementation, the first color processing mode or the third color processing mode includes color retention or color enhancement.

According to the first aspect or the second aspect, in an exemplary implementation, the second color processing mode or the fourth color processing mode includes a black and white, dim, blur, or retro mode.

According to a third aspect, an embodiment of the present disclosure provides a terminal device including a camera, a memory, a processor, and a bus. The camera, memory and processor are connected via a bus. The camera is configured to capture an image and the memory is configured to store a computer program and instructions. The processor is configured to call the computer program, the instruction, and the captured image that is stored in the memory, and is further specifically configured to allow the terminal device to perform any of the foregoing possible methods.

According to a third aspect, in a possible implementation, the terminal further includes an antenna system. The antenna system receives and transmits a wireless communication signal under the control of a processor for realizing wireless communication with a mobile communication network. The mobile communication network includes one or more of the following: GSM network, CDMA network, 3G network, 4G network, 5G network, FDMA network, TDMA network, PDC network, TACS network, AMPS network, WCDMA network, TDSCDMA network, Wi-Fi network and LTE network.

The technical solutions in the above possible implementations can be combined within the scope of the present description.

In conventional image photography technology, there is no way to obtain a special effect without distinguishing individual objects or colors in any image. According to the present invention, different areas in an image can be distinguished from each other by using color to provide a special photo or video effect, and the main object and target in the image can be more clearly distinguished. Thus, the main objects are more visible. Additionally, the present disclosure may further provide more color changes and key object changes to enhance the user experience.

Brief description of the drawings

Fig. 1 is a structure diagram of a terminal device according to an embodiment of the present disclosure;

Fig. 2 is a flowchart of an image processing method according to an embodiment of the present description;

Fig. 3 shows an example of a mask identifier according to an embodiment of the present description;

Fig. 4 shows another example of a mask identifier according to an embodiment of the present description;

Fig. 5 is a target mask determination diagram according to an embodiment of the present description;

Fig. 6 is another diagram for determining a target mask according to an embodiment of the present description;

Fig. 7 is another diagram for determining a target mask according to an embodiment of the present description;

Fig. 8 is another diagram for determining a target mask according to an embodiment of the present description;

Fig. 9 is a diagram of an image processing apparatus according to an embodiment of the present disclosure; And

Fig. 10 is another diagram of an image processing apparatus according to an embodiment of the present description.

Implementation of the invention

The following is a clear and complete description of the technical solutions in the embodiments of the present description with reference to the accompanying drawings in the embodiments of the present description. Certainly, the described embodiments are merely some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by a person skilled in the art, based on embodiments of the present description without creative efforts, should be within the scope of the present description.

In embodiments of the present disclosure, a terminal device may be a device that provides a user with a video capture function and/or a data connection, a portable device having a wireless connection function, or other processing device connected to a wireless modem, such as a digital camera, single lens reflex camera, mobile phone (or called "cell" phone) or smart phone. The terminal device may be a handheld, handheld, handheld or wearable device (e.g., smartwatch), tablet computer, personal computer (PC), PDA (personal digital assistant), vehicle-mounted computer, drone, aircraft, or the like.

Fig. 1 is a diagram of a further possible hardware structure of terminal 100.

As shown in FIG. 1, terminal device 100 may include components such as RF unit 110, memory 120, input unit 130, display unit 140, camera 150, audio circuit 160 (including speaker 161 and microphone 162), processor 170, external interface 180 and power supply 190. One skilled in the art can appreciate that FIG. 1 is just an example of an intelligent terminal or multifunctional device and is not a limitation of the intelligent terminal or multifunctional device. The smart terminal or multifunctional device may include more or fewer components than those shown in the drawing, or may combine some components or include different components. For example, an intelligent terminal or multifunctional device includes at least a memory 120, a processor 170, and a camera 150.

The camera 150 is configured to capture images or video, and can be used according to the instruction of an application program to realize a photographing function or a video capturing function. The camera may include components such as an imaging lens, a light filter, and an image sensor. Light rays emitted or reflected by an object enter the imaging lens, pass through a light filter, and finally converge on the image sensor. The imaging lens is generally configured to ensure convergence in the image of light emitted or reflected by all objects (which may also be referred to as the scene being photographed, the objects being photographed, the target scenario or targets, and may also be referred to as the image scenario that the user is taking pictures) in the viewing angle of taking pictures. The light filter is generally configured to filter out an excess light wave (eg, a light wave other than visible light, such as infrared light) in the light beams. The image sensor is mainly configured to: perform opto-electrical conversion of the received optical signal, convert the optical signal into an electrical signal, and supply the electrical signal to the processor 170 for further processing. The camera may be located on the front of the terminal or may be located on the back of the terminal. The specific number and specific arrangement of chambers can be flexibly determined based on the requirement of the designer or supplier. This is not limited in the present invention.

Input block 130 may be configured to receive input digital or character information and generate a control input related to user settings and control functions of the multifunction device. In particular, the input block 130 may include a touch screen 131 and/or other input device 132. The touch screen 131 can sense a touch operation (e.g., an operation performed by a user on or near the touch screen using any suitable object, such as a finger, connection, or stylus) of the user on or near the touch screen 131 and control the corresponding connected devices on based on the given program. The touch screen may detect a user's touch action on the touch screen, convert the touch action into a touch signal, send the touch signal to processor 170, and may receive and execute a command sent by processor 170. The touch signal includes at least contact coordinate information. The touch screen 131 may provide an input interface and an output interface between the terminal device 100 and the user. Further, the touch screen may be implemented in various types such as resistive type, capacitive type, infrared type, and surface acoustic wave type. In addition to the touch screen 131, the input unit 130 may further include another input device. In particular, other input device 132 may include, but is not limited to, one or more physical keyboards, a function key (eg, a volume control key or an on/off key 133), a trackball, a mouse, a joystick, and the like.

The display unit 140 may be configured to display information entered by the user or information provided to the user, various menus of the terminal device 100, an interaction interface, a file, and/or playback of any media file. In this embodiment of the present description, the display unit is further configured to display an image/video captured by the terminal using the camera 150. The image/video may include a preview image/video in some photographic modes, a photographed start image/video, and a target image/video. image/video on which a specific processing algorithm is performed after photographing.

In addition, the touch screen 131 may cover the display panel 141 . Upon detecting a touch operation on or near the touch screen 131, the touch screen 131 sends the touch operation data to the processor 170 to determine the type of touch event. The processor 170 then provides the appropriate visual output on the display panel 141 based on the type of touch event. In this embodiment, the touch screen and the display unit can be integrated into one component to implement the input, output, and display functions of the terminal device 100. For ease of description, in this embodiment of the present description, the touch display screen is a combination of the touch screen and the display unit. In some embodiments, the touch screen and the display unit may alternatively be used as two independent components.

Memory 120 may be configured to store instruction and data. The memory 120 may generally include an instruction storage area and a data storage area. The data storage area can store data such as media file and text. The instruction storage area may store program units such as an operating system, an application, and an instruction required by at least one function, a subset thereof, or an extension thereof. Memory 120 may further include non-volatile random access memory and provide processor 170 with functions including managing hardware, software, and data resources in the computing processing device and supporting software and application management. The memory 120 is further configured to store the media file and store the execution program and the application.

The processor 170 is the control center of the terminal device 100 and is connected to various parts of the entire terminal device through various interfaces and communication lines. The processor 170 performs various functions of the terminal device 100 and processes data by executing an instruction stored in the memory 120 and calls the data stored in the memory 120 to perform general control of the terminal device. Alternatively, processor 170 may include one or more processing units. Preferably, processor 170 may be integrated with an application processor and a modem processor. The application processor mainly processes an operating system, a user interface, an application program, and the like. The modem processor primarily handles wireless communication data. Obviously, the modem processor may alternatively not be integrated into the processor 170. In some embodiments, the processor and memory may alternatively be implemented on the same chip. In some embodiments, the processor and memory may respectively be implemented on independent chips. Processor 170 may be further configured to: generate an appropriate control control signal, send a control signal to an appropriate component in the computing processing device, and read and process data in software, in particular, read and process data and program in memory 120 so that functional modules in processor 170 perform respective functions to control respective components to perform actions as required by the instructions.

RF unit 110 may be configured to receive and send information or receive and send a signal during a call. For example, the RF unit 110 receives the downlink information from the base station, then delivers the downlink information to the processor 170 for processing, and transmits the associated uplink data to the base station. Typically, an RF unit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. Additionally, the RF unit 110 may further communicate with the network device and the other device via wireless communication. Wireless communication may use any communication standard or protocol including, but not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), Email, Short Message Service (SMS) etc.

Audio circuit 160, speaker 161, and microphone 162 may provide an audio interface between the user and terminal device 100. Audio circuit 160 may provide speaker 161 with an electrical signal converted from the received audio data, and speaker 161 converts the electrical signal into an audio signal for output. Additionally, the microphone 162 is configured to collect an audio signal and can further convert the collected audio signal into an electrical signal. Audio circuit 160 receives an electrical signal, converts the electrical signal into audio data, outputs the audio data to processor 170 for processing, and then transmits the processed audio data to, for example, another terminal via RF unit 110, or outputs the processed audio data to memory 120 for further processing. The audio circuitry may also include a headset jack 163 configured to provide a connection interface between the audio circuitry and a headset. The specific number and specific arrangements of speakers and microphones can be flexibly determined based on the requirement of the designer or supplier. This is not limited in the present invention.

Terminal 100 further includes a power source 190 (eg, a battery) that provides power to each component. Preferably, a power supply may be logically connected to the processor 170 using a power management system to perform functions such as charging, unloading, and power management using the power management system.

Terminal device 100 further includes an external interface 180. The external interface may be a standard MICRO-USB port or may be a multi-pin connector. The external interface may be configured to connect terminal device 100 to another device for communication, or may be configured to connect to a charger to charge terminal device 100.

Although not shown, terminal device 100 may further include a flash, a wireless fidelity (Wi-Fi) module, a Bluetooth module, sensors with various functions, and the like. Details are not described. Some or all of the methods described below may be applied to the terminal shown in FIG. 1.

The present description may be applied to a terminal device having a photographing function (including at least one of a photographing function or a video capturing function), and the device may be implemented in the form of an intelligent terminal device, such as a device on which a camera is installed. such as mobile phone, tablet, DV, camcorder, camera, laptop, laptop, smart robot, TV, security system or drone. In particular, the functional modules of the present invention may be deployed on a DSP chip of an appropriate device, and may be specifically an application program or software on a DSP chip. In the present invention, functional modules are deployed on a terminal device to provide an image processing function by installing software or updating and using hardware.

The present description mainly applies to a scenario in which an image or video is taken by a terminal device. Users are increasingly demanding video shooting and get a special video processing effect while taking photos to convey emotions when shooting video "What you see is what you get." According to the present invention, segmentation of the main object in an image or video can be performed, and the colors of various areas can be adjusted to realize a special real-time image effect.

Hereinafter, the present description will be described by way of examples.

Example 1

A detailed description is given with reference to FIG. 2. FIG. 2 is a flowchart of an image processing method according to an embodiment of the present disclosure. The method is performed in the process of photographing the image. In a specific implementation process, the terminal device may configure the photographing mode. In the photographing mode, the method may include the following steps.

Step 21: Acquire (which can also be understood as a photograph or capture) an image.

In particular, when a user takes a photo of an image, the corresponding preview stream is also displayed on the screen. The preview image can typically refer to one image in the preview stream. When the shutter is pressed, the user receives the photographed image. The size of the photographed image, for example, is equal to, but not limited to, 1920 x 1080.

Step 22: determine the target area and the background area in the image based on the content (which can be considered as scene semantics) in the photographed image. More specifically, the target area and the background area may be determined in the image based on the category of the object in the image. The background area is an area of the image other than the target area. The target area corresponds to a target object in the image, namely, an object that the user highlights in the image, and may be associated with an interactive user selection or system setting. In particular, step 22 may include S221 to S224.

S221: pre-process the image.

The original size photographed image is downsampled and converted to a lower resolution image. When performing calculations, the calculation amount can be reduced based on the small image. In a specific implementation process, the original size (eg, m0 x n0) may be downsampled to size m x n. Smaller values of m and n indicate fewer subsequent calculations. However, if the values of m and n are excessively small, the pixel resolution is subsequently reduced. Experiments have shown that the corresponding ranges for m and n are [128, 512] and more specifically [256, 300]. Additionally, m and n may be equal or unequal. For example, a 1920x1080 image may be downsampled to 256x256.

S222: input the image mxn downsampling image into the neural network to perform semantic segmentation to determine the image mask (Mask).

Semantic segmentation refers to pixel-level segmentation performed on an object in an image, and the category of the object is marked for each pixel. An area without an object category is marked as "background".

In particular, semantic segmentation can use a deep learning algorithm based on CNN (Convolutional Neural Network). The network model based on CNN is specifically described as follows:

(1) downsampling and convolution is performed on the mxn image to obtain the m1 x n1 image, the m2 x n2 image, ... and the mz x nz image, and the semantic features of the image are layer-by-layer extracted to obtain the m1 x n1 feature map, m2 x n2 feature map, … and mz x nz feature maps, namely, multi-scale semantic features, where m1, m2, … and mz are in multiple relationships and less than m, and n1, n2, …, nZ are in multiple relationships and less than n. For example, m=2m1=4m2=, …, =2 ^z xmz, and n=2n1=4n2=, …, =2 ^z xnz. The z value and multiple relationships can be determined based on the performance of the algorithm and the structure requirement.

(2) on m1xn1 feature map, m2xn2 feature map, ... and mzxnz feature map, upsampling and convolution are performed to combine multi-scale semantic features.

The above-mentioned convolution, downsampling and upsampling may use techniques well known in the art and are not limited and listed in the present invention.

(3) determine the object category to be identified in the image, calculate the score of each object category at each pixel, the object category (which can be a category for short) with the highest score is used as the classification result of the pixel, and finally obtain the mask graph, and Namely, the mask.

For example, if the terminal can identify categories of k objects (eg, at least one of a person, animal, plant, other preset object, or background), then k images can be obtained. Each pixel in the image receives a score that belongs to the category of the object. A higher score indicates a higher probability that the pixel belongs to the feature category.

After determining the object category of any pixel, the pixel can be identified. For example, 1 is used to represent a person, 2 is used to represent a vehicle, 3 is used to represent an animal, 4 is used to represent a plant, and 0 is used to represent a background. This is just an example and does not represent any limitation. The user can arbitrarily set the number of categories, categories, and identification method based on requirements. In FIG. 3 shows a specific example. All pixel areas in which the vehicle is located are classified as a vehicle by the neural network and identified as 1. All pixel areas in the surrounding background are classified as background by the neural network and identified as 0. For another example, in the mask output by the neural network, areas of objects of the same category have the same designation. For example, the background label is 0, the cat label is 1, and the skateboard label is 2. In the mask shown in FIG. 4, the same color may additionally be used to represent designations of the same object category. For example, a person, a horse, and a background are respectively identified with different colors.

The mask is the result of running the semantic segmentation algorithm. In an image, all pixels that belong to a category of objects are denoted by either a color or ID, and the background is also denoted by a color or ID. The image obtained after processing is referred to as a mask, so that the segmentation result is displayed intuitively.

The content of an image may include a main subject and a background. For ease of description, respectively, the image mask may include a main object mask and a background mask. The main object mask may correspond to the main object identified using the segmentation method, including an object that the user highlights in the image or in the photographed image, for example, a person, animal, plant, or a specific object (cup, table, clothes, decoration ...). The background mask corresponds to another area that is in the image and is not identified as the mask of the main object. The image mask matches the entire image. The ability to identify the mask of the main object is related to the functioning of the neural network. For example, some neural networks can only identify a person and a background. Some neural networks can identify a person, a vehicle, and a background. Some neural networks can only identify the vehicle and the background. Some neural networks can identify a person, an animal, and a background. Some neural networks can only identify the animal and the background. Some neural networks can identify an animal, a plant, a background...

It should be understood that the image may alternatively include only the main subject, or may include only the background. When the image only includes the main object, the main object can also be identified as the background. These settings for the main subject and background in the image can be flexibly designed and defined by the designer.

A large amount of segmentation training data needs to be used to train a deep neural network, and the training data set includes a large number of images, including segmentation object categories, input images, and mask graphs. The training dataset can cover various typical segmentation object application scenarios and has a variety of data. The input images and mask graphs in the training dataset are used to train the network to obtain an excellent network parameter, in other words, to obtain segmentation performance that satisfies the user. The resulting network parameter is used as the final used neural network calculation parameter.

S223: determine the target mask based on the masks.

Different masks can be obtained for different images and neural networks with different capabilities. The terminal may further define a mask, which is within the masks, and which corresponds to the object to be highlighted and prominently displayed. That is, the target mask must be defined. Determining the target mask includes, but is not limited to, the following several methods.

Method 1: If the mask only includes one main object mask and one background mask, then the main object mask is defined as the target mask.

In particular, it is assumed that segmentation is performed to obtain k masks on the image. K masks correspond to different categories of the object. If k=2 and the two masks include one object mask and one background mask, then the image area corresponding to the object mask is defined as the target area, and the area corresponding to the background mask is defined as the background area.

As shown in FIG. 5, the image masks output by the neural network only include the main object mask A1 and the background mask. In this case, A1 can be defined as the target mask.

Method 2: when the mask includes a plurality of main object masks and a background mask, if the number of pixels included in any main object mask exceeds a specific threshold value, then the main object mask is determined as the target main object; or, if the number of pixels included in any main object mask is less than a specific threshold value, then the main object mask is redesignated and also designated as background. The number of pixels included in the main object mask may be the number of pixels included in the area of a single object in the image.

In particular, it is assumed that semantic segmentation is performed to obtain k masks on the image. k masks correspond to different categories of the object. If k is greater than 2, and the number of pixels included in the k0 masks in the k masks exceeds a predetermined threshold value, then the image area corresponding to the k0 object masks is determined as the target area, and the image area corresponding to the remaining mask is determined as the background area, where k0 is a non-negative integer less than k.

As shown in FIG. 6, the neural network image output masks include a main object mask A1, a main object mask A2, and a background mask. If the number of pixels included in A1 is greater than the predetermined threshold and the number of pixels included in A2 is less than the predetermined threshold, A1 is determined as the target mask, and the main object mask A2 is redesignated as the background mask. In FIG. 5 shows the redesignated mask. If the number of pixels included in A1 is greater than the predetermined threshold, and the number of pixels included in A2 is also greater than the predetermined threshold, both A1 and A2 are determined as the target mask. If neither the number of pixels included in A1 nor the number of pixels included in A2 exceeds the predetermined threshold value, A1 and A2 are re-identified as background masks. In other words, the image does not include the mask of the main object.

It should be understood that in a particular implementation process, A1 and A2 may have the same object category or different object categories.

Method 3: When the mask includes a plurality of main object masks and a background mask, the main object mask including the largest number of pixels is selected as the target mask, and other main object masks are also re-designated as the background mask.

In particular, it is assumed that semantic segmentation is performed to obtain k masks on the image. k masks correspond to different categories of the object. If k is greater than 2, the image area corresponding to the mask that includes the largest number of pixels, and that is, in k masks, is determined as the target area, and the image area corresponding to the remaining masks is determined as the background area.

As shown in FIG. 6, the neural network image output masks include a main object mask A1, a main object mask A2, and a background mask. A1, which includes the largest number of pixels, is designated as the target mask, and the main object mask A2 is re-designated as the background mask. In FIG. 5 shows the redesignated mask.

It should be understood that in a particular implementation process, A1 and A2 may have the same object category or different object categories.

Method 4: When the mask includes a plurality of main object masks and a background mask, and the plurality of main object masks includes a plurality of object categories, the target mask is determined based on object category priorities. For example, if the priority of the person's mask is higher than the priority of the vehicle's mask, then the person's mask is the target mask, and the vehicle's mask may be redesignated as the background. For example, if the priority of a human mask is higher than the priority of an animal mask and is higher than the priority of a plant mask, and the priority set by the system is that all masks whose priorities are higher than the priority of a plant mask are masks of the main object, then, both the human mask and the animal mask are target masks, and the plant mask can be redesignated as the background. It should be understood that one or more items belong to the same object category mask.

In particular, it is assumed that semantic segmentation is performed to obtain k masks on the image. k masks correspond to different categories of the object. If k is greater than 2, the target mask is determined from the k masks based on the given object category priorities. The area of the image corresponding to the target mask is defined as the target area, and the area of the image corresponding to the remaining masks is defined as the background area.

As shown in FIG. 7, the masks of the image output by the neural network include a main object mask A1, a main object mask B1, and a background mask. A1 and B1 have different object categories, and A1's priority is higher than B1's. If the system specifies that any base object mask whose priority is higher than or equal to that of B1 can be used as the target mask, both A1 and B1 are target masks. If the system determines that the main object mask whose priority is higher than B1 can be used as the target mask, A1 is determined as the target mask, and B1 is redesignated as the background mask.

Method 5: If the mask includes a plurality of main object masks and a background mask, the target mask may be determined according to a user selection operation. The input mode includes, but is not limited to, selection instructions such as screen touch and voice. The main object mask corresponding to the item selected by the user is the target mask.

In particular, it is assumed that semantic segmentation is performed to obtain k masks on the image. k masks correspond to different categories of the object. If k is greater than 2, the target mask is determined from the k masks according to the user's selection instruction. The area of the image corresponding to the target mask is defined as the target area, and the area of the image corresponding to the remaining masks is defined as the background area.

As shown in FIG. 7, the image masks output by the neural network include a main object mask A1, a main object mask B1, and a background mask. If the user's input during photography matches A1 on the touch screen, then A1 is defined as the target mask and B1 is redesignated as the background mask. If the user's input during photography matches B1 on the touch screen, then B1 is defined as the target mask and A1 is redesignated as the background mask.

Method 6: If the mask includes a plurality of main object masks and a background mask, and the plurality of main object masks includes a plurality of object categories, a target mask may be determined based on input of a selection operation by a user. The input mode includes, but is not limited to, selection instructions such as screen touch and voice. All masks of the main objects of the object category corresponding to the subject selected by the user are target masks.

In particular, it is assumed that semantic segmentation is performed to obtain k masks on the image. k masks correspond to different categories of the object. If k is greater than 2, the target mask is determined from the k masks according to the user's selection instruction. The area of the image corresponding to the target mask is defined as the target area, and the area of the image corresponding to the remaining masks is defined as the background area.

As shown in FIG. 8, the image masks output by the neural network include the main object masks A1, A2, B1 and B2 and the background mask. A1 and A2 have the same object category, B1 and B2 have the same object category. If the user's input during photography matches A1 on the touch screen, A1 and A2, which have the same subject category, are defined as target masks, and B1 and B2 are redesignated as background masks. If the user's input during photography matches B2 on the touch screen, B1 and B2 of the same object category are defined as target masks, and A1 and A2 are redesignated as background masks.

It should be understood that the above specific implementations or embodiments are merely examples and should not be limiting. The above specific implementations or embodiments can be freely combined without breaking the logic. Therefore, after performing mask segmentation on an image, one or more target masks can be obtained. These target masks may have one or more object categories, and each target mask object category may further include one or more individual items. The result refers to a rule that is set in the terminal device to determine the target mask and user input. In some scenarios, the image may alternatively include only the background mask.

S224: determine the target area and the background area in the source image.

The original size photographed image is sampled from the mask, and the target mask and background mask in the mask are also upsampled. The area composed by all pixels which are the upsampled target mask and which correspond to the source image is the target area, and the area composed by all pixels which are the upsampled background mask and which correspond to the source image is the background area .

Step 23: Process the target area and the background area on the image in different color processing modes to obtain the target image. Various color processing modes are used for processing, so that the chroma of the target area is greater than the chroma of the background region, or the luminance of the target region is greater than the luminance of the background region. In other words, the chroma of the target area in the target image is greater than the chroma of the background area in the target image, or the luminance of the target area in the target image is greater than the luminance of the background region in the target image.

In particular, the first color processing mode and the second color processing mode are respectively used for the target area and the background area in the image. The first color processing mode and the second color processing mode include, but are not limited to, the following methods.

Method 1: The first color processing mode preserves the color, and the second color processing mode uses a filter, such as converting the color of the background area to black and white. Typical filters further include any black and white filter, blackout filter, retro filter, film filter, blur filter, bokeh filter, and the like.

For example, the black and white filtering operation is to map each pixel value to a gray value to realize a black and white filter effect. As another example, the dimming filter dims the brightness of each pixel value to achieve a specific dimming effect.

Method 2: The first color processing mode is the first filtering method, the second color processing mode is the second filtering method, and the first filtering method is different from the second filtering method. For the same image, the color of the image obtained in the first filtering method is greater than the color of the image obtained in the second filtering method.

Method 3: The first color processing mode is the third filtering method, the second color processing mode is the fourth filtering method, and the third filtering method is different from the fourth filtering method. For the same image, the brightness of the image obtained in the third filtering method is greater than the brightness of the image obtained in the fourth filtering method.

It should be understood that color is represented by both luma and chrominance. Chroma is a property of a color that does not include luminosity and reflects the hue and saturation of a color, and luminosity refers to the brightness of a color. Therefore, color processing includes luma processing and/or chrominance processing.

In particular, the filter may include chroma, brightness, and hue adjustment, and may further include texture mapping and the like. The color system can be adjusted in the target method by adjusting the hue and hue so that the color system becomes richer or paler, or the tone can be changed and the other color system remains unchanged. A filter can also be thought of as a pixel-to-pixel mapping. The pixel value of the input image is mapped to the pixel value of the target pixel using a target mapping table to achieve a special effect. It should be understood that the filter may be a given mask parameter. These color-related parameters may be parameters in the filter mask that are well known in the art, or may be parameters independently developed by the user.

Additionally, after step 23, the method further includes the step 24 of storing the image processed in step 23.

According to the present invention, in the photographing process, the terminal device can determine the target object and background based on the content of the image, and perform different color processing on the target object and the background, so that the main object in the image photographed by the user can be more prominent, and the photographed image is similar. to the film.

Example 2

Specifically, in the present invention, the image processing method for video recording is similar to the image processing method for taking photographs, and the difference is that the object processed in a photograph is one image, while the object processed in video recording is successive frames of video, namely, a set of consecutive images. The object processed in a video recording may be a completed video, or may be a segment in a completed video, or a custom video clip within a time period range. For the processing procedure for each image in a video or video clip, refer to the processing method in Example 1.

Specifically, the image processing method for capturing a video may include the following steps.

Step 31: get N photographed images, where N is a positive integer; and perform the operations of steps 32 and 33 on each picture, where the N pictures may be adjacent video frames and the sum of the N pictures may be considered as video. Alternatively, N pictures may not be adjacent.

An exemplary implementation of step 32 may be the same as step 22.

An exemplary implementation of step 33 may be the same as step 23.

Additionally, since the video includes successive images, the method of determining the subject also refers to the temporal sequence. Therefore, in addition to step 23, more operations can be performed in step 33. Alternatively, any method of determining the main object in S223 may have a delay. For example, a person and a background are defined in the L1 frame, and it can still be determined from (L1+1) image frame to (L1+L0) image frame by pixel marking and mask comparison that the person in these images is the main subject, and the area corresponding to the person in these images is target area 0. The main subject and background do not need to be determined for each frame. The moment at which the main object is determined each time may be determined by the user, or the main object may be periodically determined, for example, but not limited to every 2s or every 10s. The main object determination method each time includes, but is not limited to, six methods in S223.

Step 34: store a video composed of N images on which color processing is performed.

According to the present invention, in a process in which a user records a video, the terminal device can determine the target subject and background based on the video content, and perform different color processing on the target subject and background, so that the main subject of the video photographed by the user can be more prominent, the photographed video is as attractive as the movie and the user experience is improved.

Example 3

In the present invention, the image processing method for video recording is similar to the image processing method for taking photographs, and the difference is that the object processed in photography is one image, when the object processed in video recording is consecutive video frames, namely, a plurality of consecutive images. Therefore, for the processing procedure for each image, refer to the processing method in Example 1. In some complex video shooting scenarios, some areas in the image may not be detected correctly. If the same area designated separately as a target or background in adjacent frames, the same area is processed into different colors according to the above example of the color processing method, and changes in colors of the same area in adjacent frames cause flickering. Therefore, flicker must be determined and eliminated during processing. The flickering may be an error in determining the category of the object.

In the detection method when there is flickering video, the previous frame mask can be processed based on the optical flow to obtain an optical flow based mask, and the difference between the mask based on the optical flow and the mask of the current frame is compared. When the degree of match or degree of similarity exceeds a specific proportion, no flicker is determined. When the degree of coincidence or degree of similarity does not exceed a certain proportion, the presence of flicker is determined. Additionally, it should be understood that flicker detection is a continuous process. Alternatively, a specific way to determine if flicker is present is as follows:

(1) First, the optical flow of adjacent frames is calculated, where the optical flow indicates the offset relationship between pixels in adjacent frames ((t-1) frame and t frame).

(2) obtain the mask (t-1), calculate the optical flow mask F based on the frame mask (t-1) and the optical flow information (t-1) of the frame and t frame, where the optical flow mask is obtained by calculation based on the optical flow .

(3) get the frame mask S t .

(4) calculate the main object pixel set SF in the optical flow mask F, and calculate the main object pixel set SS in the mask S. The number of pixels in the union set and the intersection set SF and SS are Nu and Ni, respectively. When (Nu-Ni)/Nu exceeds a specific threshold, it is determined that there is a relatively large difference between the masks of neighboring frames (t-1) and t, and the presence of flicker between (t-1) frame and t frame is determined, or, obviously, flicker occurs in t frame. A relatively large difference indicates that the same object may be incorrectly assessed as belonging to different categories of objects. For example, the same object in frames (t-1) and t is separately defined as a man and a monkey.

Alternatively, if in the first N0 (positive integer greater than 2) images of the current image, the number of neighboring image groups with the same object defined as different object categories is greater than a predetermined threshold value, it can be determined that at flicker elimination processing must be performed on the current frame. If it is determined that the number of adjacent picture groups with the same object defined as different object categories does not exceed a predetermined threshold value, it may be determined that flicker elimination processing is not to be performed on the current frame.

Alternatively, for example, for a given number of historical neighbor frames or a given number of historical frames, if flicker is detected in more than half of the frames (for example, it is determined that flicker occurs in three frames of video in the first five adjacent frames of the current video frame) can be determined that flicker elimination processing should be performed on the current frame. If flicker is determined to be present in less than half of the frames (eg, flicker is determined to occur in one of the first five adjacent frames of the current video frame), it may be determined that no flicker elimination processing should be performed on the current frame.

It should be understood that the current video image can be considered as the image that is currently being recorded. A moment may be understood as a general moment in some scenarios, or may be understood as a specific moment in some scenarios, such as the last moment or the moment the user is interested in.

Specifically, the image processing method for capturing a video in this example may include the following steps.

Step 41: get N photographed images, where N is a positive integer; and perform the operations of steps 32 and 33 on each picture, where N pictures can be adjacent video frames and the sum of N pictures can be understood as video; or N images may not be adjacent frames.

Step 42: determine if the number of adjacent picture groups in which flickering occurs in the first N0 frames of the current frame (current picture) exceeds a predetermined threshold value. Here N0 and the threshold value can be set by the user. For example, N0 is a selected number of historical samples of a video frame, and the threshold may be 1/2, 2/3, or the like of N0. This is just an example and is not limited to this.

If the result of the determination is not exceeding the predetermined threshold, then the operations of steps 43 and 44 are performed on the currently photographed or captured image.

An exemplary implementation of step 43 may be the same as step 32.

An exemplary implementation of step 44 may be the same as step 33.

If the determination result is above the predetermined threshold, then the operation of step 45 is performed on the currently photographed or captured image.

Step 45: process all image areas of the current frame using the same color processing method to obtain the target image. The same color processing method may be the same as the color processing method of the background area in the previous frame, or may be the same as the color processing method of the target area in the previous frame, or may be the same as the color processing method of the entire image in the previous frame. frame. For example, a color processing method that is the same as for the background area in step 33 (23) can be used for the entire image. Alternatively, a color processing method that is the same as for the target area in step 33(23) may be used for the entire image. For example, the entire image remains in color, or the entire image is black and white, or the first or second color processing mode (including but not limited to the color processing modes in Example 1) is used for the entire image.

In this case, the mask segmentation procedure similar to step 22 may be performed or omitted for the current frame. This is not limited in this example.

After step 45, step 46 of storing a video composed of N images on which color processing is performed. N is a positive integer.

According to the present invention, in a process in which a user records a video, the terminal device can determine the target subject and background based on the video content, and perform different color processing on the target subject and background, so that the main subject of the video photographed by the user can be more prominent, the photographed the video is as attractive as the movie and the user experience is improved.

Example 4

In some application scenarios, the content of the image taken by the user usually changes. Therefore, the main subject of the image usually changes. The user can also freely select the color processing mode of the main object in different images to independently control the style of the video.

The image processing method in the video shooting process may include the following steps.

Step 51: The terminal receives a video frame.

Step 52: The terminal device determines the main object area and the background area in any video frame obtained from the video.

Step 53: The terminal device uses any color processing mode at any time for the main object area, and uses any color processing mode at any time for the background region. However, it is necessary for any image to ensure that the luminance or chrominance of the main object area after color processing is greater than the luminance or chrominance of the background area after color processing. Alternatively, for any image, the chrominance or luminance of the image obtained in the color processing mode used for the main subject area was greater than the chrominance or luminance of the image obtained in the color processing mode used for the background region.

Example 5

In some usage scenarios, the content of the image taken by the user usually changes. Therefore, the main subject of the image usually changes. The user can also freely select the color processing mode of the main object in different images to independently control the style of the video. Especially, the color changes over a period of time.

The image processing method in the video shooting process may include the following steps.

Step 61: Capture N1 images in the first time period and capture N2 images in the second time period, when the first time period and the second time period are adjacent time periods and both N1 and N2 are positive integers, the first time period and the second period times may be the duration in which the user can identify the change in the image with the naked eye, and N1 and N2 are determined by the frame rate and the length of time periods during video recording. This is not limited in the present invention.

Step 62: determine the first target area and the first background area in each of the N1 images, where the first background area is an image area different from the first target area, and the first target area in each of the N1 images correspond to the first object (which may include , at least one object); and defining a second target area and a second background area in each of the N2 images, where the second background area is an image area different from the second target area, and the second target area in each of the N2 images corresponds to a second object (which may include, at least one item).

Step 63: process the first target region in the first color processing mode, process the first background region in the second color processing mode, process the second target region in the third color processing mode, and process the second background region in the fourth color processing mode to obtain a target video, where in the target video, the chroma of the first target region is greater than the chroma of the first background region, or the luminance of the first target region is greater than the luminance of the first background region; and the chroma of the second target area is greater than the chroma of the second background region, or the luminance of the second target region is greater than the luminance of the second background region.

Example 6

In some implementation scenarios, the content of the image photographed by the user usually changes. Therefore, the main subject of the image usually changes. The user is also free to choose the main target that the user wants to highlight in different images. For example, the image area corresponding to the first object is determined as the target area in the first time period, the image area corresponding to the second object is determined as the target area in the second time period, and the first object and the second object are different objects, objects, or categories of objects.

In this scenario, the image processing method in the video shooting process may include the following steps.

An additional implementation of step 71 may be the same as step 61.

Step 72: determine the first target area and the first background area in any of the N1 images based on the image content, and determine the second target area and the second background area in any of the N2 images based on the image content, where the object or category of the object corresponding to the second target area, different from the object or object category corresponding to the first target area, so that the system and the user can independently select the target main object and the target image area. The image includes a main object and a background, and accordingly, the image includes a target area and a background area.

For example, the first object is a human and the second object is an animal. For example, the first object is person A and the second object is person B. For example, the first object is two people and the second object is a dog and two cats. The remaining area, which is not identified, is designated as the background.

In this method, the image mask may be determined by using the above methods in S221 and S222. However, the following method is not limited to defining the target in the mask for each image.

Perhaps in the image mask, the user can freely input the first object and the second object, and the first object and the second object are determined according to the user's selection instruction. For example, if the user selects an item, the system identifies the pixel corresponding to the instruction entered by the user, additionally identifies a particular (/some) item(s) (/may be at least one item) or a particular (/some) category of the object ( /object category) (/may be at least one object category) of the mask selected by the user further defines a specific (/some) item(s) or all items in a specific (/some) object category (/object category) , as the first object, and defines the first object or image region corresponding to the first object as the first target region. This situation can be maintained for a certain period of time. In other words, in the next few frames, the area corresponding to the mask corresponding to the first object is the first target area until the user selects another item at the next moment, and the area corresponding to the new item is determined to be the second target area according to a method similar to the above. way. In the image, the area of the image other than the first target area or the second target area is the background area. Specifically, the area corresponding to the mask corresponding to the first object in the first time period is the first target area, and the area corresponding to the mask corresponding to the second object in the second time period is the second target area.

Alternatively, in the image mask, the system may determine the target image mask in the time period in the image mask based on a predetermined time interval (for example, but not limited to 1s or 2s) or a predetermined number of frames (for example, but not limited to 50 frames or 100 frames). For example, the first target mask is determined at the 101st frame, and a mask that has the same object category or subject as the first target mask at the 101st frame is used as the first target mask for every next 102 frames up to 200 frames up to determining the second target mask in the 201st frame. For each of the next 202 frames up to 300 frames, a mask that has the same object or subject category as the second target mask in the 201st frame is used as the second target mask. It should be understood that the number of the above example can be set by the user or the system. Specifically, a target mask is determined at a point in time, and a mask of that type or a mask of that item is continuously used over a period of time.

To describe the method for determining the first target mask belonging to the first category and the second target mask belonging to the first category without limitation, reference can be made to any of the six methods in step S223. Therefore, the first target mask and the second target mask may be of the same object category or the same object, or may be different object categories or different objects. This aspect relates to the identification capabilities of the network, changing the image of the scene or the input command of the user.

Further, the first target area, the first background area, the second target area, and the second background area are further determined according to the method in S224. Details are not described in this example.

An exemplary implementation of step 73 may be the same as step 63.

Additionally, since this example may vary depending on the time period, there may be many combinations of color processing methods.

For example, the first color processing mode is the same as the third color processing mode, and the second color processing mode is the same as the fourth color processing mode. This color processing mode has good consistency.

For example, the first color processing mode is the same as the third color processing mode, and the second color processing mode is different from the fourth color processing mode. In this processing mode, the colors of the target main object are consistent and the background colors are changed so that the overall visual effect is more stunning.

For example, the first color processing mode is different from the third color processing mode, and the second color processing mode is the same as the fourth color processing mode. In this processing mode, the background colors are consistent and the target main object colors are changed so that the main target object is more visible.

For example, the first color processing mode is different from the third color processing mode, and the second color processing mode is different from the fourth color processing mode. In this color processing mode, more color transformations can be provided and a greater level of color matching can be provided based on the requirements of different scenarios.

The first color processing mode or the third color processing mode includes a filter such as a color-holding or color-enhancing filter. The second color processing mode or the fourth color processing mode includes filters such as a black and white filter, a dark filter, a retro filter, a film filter, a blur filter, and a bokeh effect filter.

In particular, for the color processing methods for the target area and the background area of the same image, refer to step 23. For N2 images, the third color processing mode and the fourth color processing mode are respectively the same as the first color processing mode and the second color processing mode.

According to the above solutions, in some scenarios, in order to highlight different backgrounds, the user can freely select the background color processing mode in different images. In some scenarios, the user can freely select the color mode of the main object in different images in order to highlight the main object in different shapes and different proportions.

It should be understood that in different examples of the present description, the signals indicated by the same marking may have different sources or may be obtained in different ways. This is not a limitation. Additionally, references to the step of various examples "same as step XX" point more to the general signal processing logic of the two steps. It is not limited that both the input and output of the two steps must be exactly the same, and that the two method procedures are completely equivalent. Appropriate references and variations, which may be used by a person skilled in the art, should be within the protection scope of the present description.

The present description relates to an image processing method. The target area and the background area are determined on the image by performing mask segmentation on the image. Different color processing modes are applied to the target area and the background area so that the luminance of the target area is larger than the luminance of the background area, or the chroma of the target area is larger than the chroma of the background area, and the main object corresponding to the target area is more prominently highlighted. This realizes the special effect of the movie.

Based on the image processing method provided in the above embodiment, an embodiment of the present description provides an image processing apparatus 900. The device may be used in a variety of terminals and may take any form of implementation of the terminal 100, such as a terminal having a video capture function. Referring to Fig. 9, the device includes:

a photographing module 901, configured to acquire an image that can be photographed or captured a video, wherein the module is specifically configured to perform the method in step 21, step 31, step 51, step 61, or step 71 in the above example, and a method that can be equivalent to the method, and the module can be implemented by the processor by calling the appropriate program instruction in memory to control the camera to capture an image;

a determining module 902, configured to determine a target area and a background area in an image based on the content of the image, where the module is specifically configured to perform the method in step 22, step 32, step 52, step 62, or step 72 in the above example, and the method that may be equivalent to a method, and a module, may be implemented by the processor by calling the appropriate program instruction in memory to implement the appropriate algorithm; And

a color processing unit 903, configured to use different color processing modes for the target area and the background area in the image to obtain the target image or the target video, so that the chroma of the target area is greater than the chroma of the background area, or the luminance of the target area is greater than the luminance of the area background, where the module is specifically configured to execute the method in step 23, step 33, step 53, step 63, or step 73 in the above example and the method, which may be equivalent to the method, and the module may be implemented by the processor by calling the appropriate program instruction in memory using a particular algorithm.

Additionally, the device may further include a storage module 904 configured to store an image or video on which color processing is performed.

The previously described specific examples of the method, explanation and description of the technical characteristics in the embodiments and expansion of the plurality of implementation forms are also applicable to the execution of the method in the device, and the details are not described in the device embodiments.

The present description proposes an image processing device. The mask is segmented on the image so that the target area and the background area on the image are determined based on the content of the image. Different color processing modes are applied to the target area and the background area so that the luminance of the target area is larger than the luminance of the background area, or the chroma of the target area is larger than the chroma of the background area, and the main object corresponding to the target area is more prominently highlighted. This provides a special film effect.

Based on the image processing method provided in the above embodiment, the embodiment of the present description further provides an image processing apparatus 1000. The device may be used in a variety of terminals and may be any form of implementation of the terminal 100, such as a terminal having a video capture function. Referring to FIG. 10, the device includes:

a photographing module 1001 configured to acquire an image that can photograph or shoot a video, where the module is specifically configured to perform the method in step 21, step 31, step 51, step 61, or step 71 in the above example, and a method that may be equivalent to method, and the module can be implemented by the processor by calling the appropriate program instruction in memory to control the camera to capture an image;

an evaluator 1002, configured to: determine whether the number of flickering frames in the first N0 frames of the current frame exceeds a predetermined threshold value; and, if the determined result is that the number of flickering frames does not exceed a predetermined threshold value, continue to initiate the determination module 1003 and the color processing module 1004 to execute the corresponding functions; or, if the determined result is that the number of flickering frames exceeds a predetermined threshold, continue to cause the flicker removal module 1005 to execute the appropriate functions; and the module 1002 is specifically configured to execute the method in step 42 in the preceding example and the method, which may be equivalent to the method and the module may be implemented by the processor by calling the corresponding program instruction in memory to implement the corresponding algorithm;

determining module 1003, configured: when the evaluation module 1002 determines that the number of flickering frames in the first N0 frames of the current frame does not exceed a predetermined threshold value, determine the target area and the background area in the image based on the image content, where the module is specifically configured to perform the method in step 22, step 32, step 43, step 52, step 62, or step 72 in the above example, and the method, which may be equivalent to the method, and the module may be implemented by the processor by calling the corresponding program instruction in memory to implement the corresponding algorithm;

a color processing module 1004, configured to use different color processing modes for the target area and the background area in the image, so that the chroma of the target area is larger than the chroma of the background area, or the luminance of the target area is larger than the luminance of the background area, where the module is specially configured with the ability to execute the method in step 23, step 33, step 44, step 53, step 63 or step 73 in the above example, and the method, which may be equivalent to the method, and the module may be implemented by the processor by calling the appropriate program instruction in memory with a specific algorithm ; And

a flicker removal module 1005, configured: when the evaluation module 1002 determines that the number of flickering frames in the first N0 frames of the current frame exceeds a predetermined threshold value, use the same color processing method for all image areas of the current frame where the same color processing method can be the same as the way the color of the background area in the previous frame is handled, or may be the same as the way the color of the target area is handled in the previous frame; and the module is specifically configured to execute the method in step 45 in the above example and the method, which may be equivalent to the method and the module can be implemented by the processor by calling the corresponding program in memory using a particular algorithm.

Additionally, the device 1000 may further include a storage module 1006 configured to store an image or video on which color processing is performed.

The above specific example of the method, explanations and descriptions of the technical features in the embodiments and expansions of the plurality of implementation forms are also applicable to the execution of the method in the device, and the details are not described in the device embodiments.

The present description provides an image processing apparatus. The mask is segmented on the image so that the target area and the background area on the image are determined based on the content of the image. Different color processing modes are applied to the target area and the background area, so that the luminance of the target area is larger than the luminance of the background area, or the chroma of the target area is larger than the chroma of the background area, and the main object corresponding to the target area is more prominently highlighted. This realizes the special effect of the film.

It should be understood that the division into modules in the above apparatus is simply a logical functional division. In an actual implementation, some or all of the modules may be integrated into one physical unit, or may be physically separated. For example, each of the above modules may be a separate element of the processor, or may be integrated into a terminal device chip, or may be stored in a storage element of the controller in the form of program code. The processor element calls and executes the function of each of the above modules. Additionally, the modules may be integrated or may be implemented independently. The processor element herein may be an integrated circuit and has signal processing capabilities. During implementation, the steps in the preceding methods or the above modules may be implemented by integrated logic in a processor element or by instructions in the form of software. The processor element may be a general purpose processor, such as a central processing unit (CPU for short), or may be one or more integrated circuits configured to implement methods, such as one or more application-specific integrated circuits (application- specific integrated circuit, ASIC for short), one or more microprocessors (digital signal processor, DSP for short), or one or more field-programmable gate arrays (FPGA for short).

It should be understood that in the specification, claims, and accompanying drawings of this specification, the terms "first", "second", and the like are intended to distinguish between like objects, but do not necessarily indicate a particular order or sequence. It should be understood that data thus referred to are interchangeable in the proper circumstance, such that the embodiments described herein may be implemented in other orders than the order shown or described herein. Additionally, the terms "includes", "comprising" and any other variations means to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or modules, not necessarily limited to the steps or modules expressly listed. , but may include other steps or modules not expressly stated or inherent in such a process, method, product, or device.

Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as a method, system, or computer software product. Therefore, the present description may take the form of hardware-only embodiments, software-only embodiments, or combination of software and hardware embodiments. Moreover, the present disclosure may take the form of a computer program product that is embodied in one or more computer-readable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, and the like) that includes a computer program code.

The present description has been described with reference to flowcharts and/or flowcharts of a method, apparatus (system), and computer software product in accordance with embodiments of the present disclosure. It should be understood that computer program instructions may be used to implement each process and/or each block in the flowcharts and/or flowcharts and the combination of the process and/or block in the flowcharts and/or flowcharts. These computer program instructions may be provided to a general purpose computer, a special purpose computer, an embedded processor, or a processor of another programmable data processing device to generate a machine, such that instructions executed by the computer or processor of another programmable data processing device generate a device to implement a particular function in one or multiple processes in flowcharts and/or one or more blocks in flowcharts.

These computer program instructions may be stored in computer readable memory, which may instruct the computer or other programmable processing device to operate in a certain way, such that the instructions stored in computer readable memory generate an artifact that includes the learning device. The learning device implements a specific function in one or more processes in the flowcharts and/or in one or more blocks in the flowcharts.

These computer program instructions may alternatively be downloaded to a computer or other programmable processing device such that a series of operations and steps are performed on the computer or other programmable device, thereby performing computer processing. Therefore, instructions executed on a computer or other programmable device provide steps for implementing a particular function in one or more processes in flowcharts and/or in one or more blocks in flowcharts.

Although some embodiments of the present disclosure have been described, changes and modifications to these embodiments may be made by those skilled in the art once they learn the basic concept of the invention. Therefore, the appended claims cover all listed embodiments and all changes and modifications should be within the scope of the present description. Obviously, a person skilled in the art can make various modifications and variations in the embodiments of the present description, without departing from the spirit and scope of the embodiments of the present description. The present description covers these modifications and variations provided that they are within the scope of protection defined by the following claims and their equivalent technologies.

Claims (34)

1. An image processing method implemented by a terminal device, comprising the following steps: capture an image; determining a main object area and a background area in the image based on the category of the object in the image and the neural network; maintaining the color of the main object area in the image, and performing black and white color processing or blur processing on the background area in the image, the background area being an image area different from the main object area; or determining only a background area in the image based on the category of the object in the image and the neural network, and performing black-and-white color processing or blur processing in the background area of the image, the background area being an image area different from the main object area; generating a target image or a target video based on the processed image; And the first mask of the first category of objects and the second mask of the second category of objects are determined based on the neural network, the first mask defines the area of the main object, and the second mask defines the background area, the first category of the object contains at least one of: human, animal, plant, vehicle means or other specified category of objects; the second category of the object is the background. 2. The method of claim 1, wherein the neural network is trained using a training dataset, wherein the training dataset contains input images. 3. The method of claim 2, wherein the training data set further comprises mask graphs. 4. The method according to any one of claims 1 to 3, wherein the neural network is configured to determine at least two categories of objects from a person, animal, plant, vehicle, or clothing. 5. The method according to any one of claims 1 to 4, wherein the first number of pixels in the image and corresponding to the first mask is greater than the second number of pixels in the image and corresponding to the third mask of the third category of objects, and the third category of the object is different from the second category of objects. 6. A method according to any one of claims 1 to 5, wherein the first mask has a higher primary object determination priority than the second mask. 7. A method according to any one of claims 1 to 6, wherein the main object area comprises a plurality of individual items, the plurality of individual items belonging to the same object category or different object categories. 8. The method according to any one of claims 1 to 7, further comprising obtaining an area of the main object by performing pixel level segmentation on the object using a neural network. 9. An image processing device used by the terminal device, comprising: a photographing module, a determination module and a color processing module, while the photographing module is configured to capture an image; the determining module is configured to identify the main object area and the background area in the image based on the category of objects in the image and use the neural network; And the color processing module is configured to: storing the color of the main object area in the image, and performing black and white color processing or blur processing in the background area of the image, the background area being an image area different from the main object area; or the determining module is configured to determine only the background area in the image based on the category of the object in the image and using the neural network, and the color processing module is configured to perform black and white color processing or blur processing in the background area of the image, and the background area is an image area different from the area of the main object; the color processing module is further configured to generate a target image or a target video based on the processed image; and the determining module is further configured to determine the first mask of the first category of the object and the second mask of the second category of the object, and the first mask defines the main object area, and the second mask defines the background area; wherein the first category of objects comprises at least one of a human, animal, plant, or vehicle; the second category of objects is the background. 10. The apparatus of claim 9, wherein the neural network is trained using the training dataset, the training dataset comprising input images. 11. The apparatus of claim 10, wherein the training data set further comprises mask graphs. 12. An apparatus according to any one of claims 9 to 11, wherein the neural network is configured to identify at least two categories of objects from a person, animal, plant, vehicle, or clothing. 13. The device according to any one of claims 9 to 12, wherein the first number of pixels in the image corresponding to the first mask is greater than the second number of pixels in the image corresponding to the third mask of the third category of objects, and the third category of objects is different from the second category of objects. 14. An apparatus according to any one of claims 9 to 13, wherein the first mask of the first category of objects has a higher primary object determination priority than the second mask of the second category of objects. 15. An apparatus according to any one of claims 9 to 14, wherein the main object area comprises a plurality of individual items, the plurality of individual items belonging to the same object category or different object categories. 16. An apparatus according to any one of claims 9 to 15, wherein the determining module is further configured to obtain an area of the main object by performing pixel level segmentation on the object using a neural network. 17. Terminal image processing device containing a camera, memory, processor and bus; wherein the camera, memory, and processor are connected via a bus; the camera is configured to capture an image; the memory is configured to store a computer program and instructions; A the processor is configured to call the computer program, instructions and the captured image stored in the memory to perform the method according to any one of claims 1-8. 18. The image processing terminal according to claim 17, further comprising an antenna system, the antenna system being configured to receive and transmit a wireless communication signal under control of the processor to implement wireless communication with a mobile communication network, wherein the mobile communication network comprises one or more of : Global System for Mobile Communications (GSM) networks, Code Division Multiple Access (CDMA) networks, 3rd generation networks (3G), 4th generation networks (4G), 5th generation networks (5G), multiple access networks frequency division multiple access (FDMA), time division multiple access (TDMA) networks, personal digital cellular (PDC) networks, all access communication system (TACS) networks, advanced mobile telephone service (AMPS) networks, broadband multiple access networks code division multiple access (WCDMA), synchronous time and frequency division multiple access (TDSCDMA), wireless reliability (Wi-Fi), or Long Term Evolution (LTE) networks.

Images