CN106898005B

CN106898005B - Method, device and terminal for realizing interactive image segmentation

Info

Publication number: CN106898005B
Application number: CN201710005330.3A
Authority: CN
Inventors: 梁舟
Original assignee: Nubia Technology Co Ltd
Current assignee: Nubia Technology Co Ltd
Priority date: 2017-01-04
Filing date: 2017-01-04
Publication date: 2020-07-17
Anticipated expiration: 2037-01-04
Also published as: CN106898005A

Abstract

A method, a device and a terminal for realizing interactive image segmentation are disclosed. The method comprises the following steps: constructing a geometric figure containing a smearing track, expanding the geometric figure to form a marking area, and determining a region of interest containing the marking area; generating an input mask map for an image segmentation algorithm: all pixels in the marked area are used as foreground points in the mask image, and pixels outside the marked area in the interested area are used as background points in the mask image; determining segmentation parameters of each pixel according to the depth map and the mask map, constructing an undirected graph, mapping the segmentation parameters of each pixel in the mask map into the undirected graph, operating a minimum segmentation-maximum flow algorithm according to the undirected graph to obtain a finely segmented mask map, and segmenting an image corresponding to a foreground point in the finely segmented mask map from an original image. The method can shorten the running time of the algorithm and improve the image segmentation effect by using the depth information of the image.

Description

Method, device and terminal for realizing interactive image segmentation

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method, an apparatus, and a terminal for implementing interactive image segmentation.

Background

Image segmentation refers to dividing a planar image into a plurality of regions that are not connected to each other according to characteristics such as color, texture, shape, and the like, which is a practical basic technology in the field of image processing. Conventional image segmentation techniques include a threshold-based segmentation method, an edge-based segmentation method, a region-based segmentation method, an energy functional-based segmentation method, a graph theory-based segmentation method, and the like. Among them, the GraphCut algorithm and its improved version, the GrabCut algorithm, are well known in the graph theory method.

The GraphCut algorithm and the GrabCut algorithm of an improved version thereof are an interactive image segmentation method based on region labeling. The GraphCut algorithm is an algorithm based on a Markov Random Field (MRF) energy minimization framework, and has the advantage that global optimal solution can be performed by combining various theoretical knowledge. The GrabCut algorithm is an improvement on the GraphCut algorithm, and is characterized in that a mask map is generated by marking foreground points (points on a target object to be extracted) and background points on an original image, a Gaussian Mixture Model (GMM) is established for foreground and background color spaces by using the original image and the mask map, energy minimization is completed by using an iterative algorithm capable of evolving in the GMM parameter learning and estimating processes, the foreground points and the background points in the image are determined, and the target image consisting of foreground point pixels is extracted from the original image.

When the GrabCut algorithm is used for image segmentation on a mobile phone, in order to reduce the complexity of interaction, strict requirements on how to mark a user are generally not made, so that the number of iterations is possibly large under the condition that the foreground points marked by the user are few, the running time of the algorithm is long, and the user experience is influenced. On the other hand, the GrabCut algorithm in the related art performs image segmentation based on a color image, and when the color feature of the target object to be extracted is not obvious, the segmentation effect by using the color image is not ideal.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a method, a device and a terminal for realizing interactive image segmentation, which can shorten the running time of an algorithm and improve the image segmentation effect by using the depth information of an image.

The embodiment of the invention provides a method for realizing interactive image segmentation, which comprises the following steps:

after detecting a smearing track on an original image, constructing a geometric figure containing the smearing track, expanding the geometric figure to form a marking area, and determining an area containing the smearing track as an interested area, wherein the interested area contains the marking area; generating an input mask map for an image segmentation algorithm: all pixels in the marked area are used as foreground points in the mask image, and pixels outside the marked area in the interested area are used as background points in the mask image;

acquiring a depth map containing depth information of a target object, and determining segmentation parameters of each pixel on a mask map according to the depth map and the mask map, wherein the segmentation parameters are used for representing the probability that the pixel is judged as a foreground point or a background point and the depth value difference between the pixel and an adjacent pixel;

constructing an undirected graph, mapping the segmentation parameters of each pixel in the mask graph into the undirected graph, and processing the undirected graph according to a minimum segmentation-maximum flow algorithm to obtain a finely segmented mask graph;

and segmenting an image corresponding to the foreground point in the finely segmented mask image from an original image obtained by shooting the target object.

Optionally, determining a segmentation parameter of each pixel on the mask map according to the depth map and the mask map includes: performing Gaussian Mixture Model (GMM) calculation according to an EM (effective electromagnetic) method, wherein the EM method comprises an E step and an M step; e and M steps of iterative operation, stopping the iterative process after the iterative operation reaches a convergence condition; determining the classification of the pixel obtained by the last M steps as the classification of the pixel, and determining the maximum probability value P of the pixel belonging to a cluster obtained by the last M steps_maxDetermining a region item segmentation parameter of the pixel, wherein the region item segmentation parameter is the probability that the pixel is judged to be a foreground point or a background point;

wherein, the step E and the step M respectively comprise the following processing:

e, step E: clustering the pixels of the same type into one or more clusters according to the depth values of the pixels on the mask map and the position relation among the pixels, and determining a GMM model of each cluster; wherein the classification of the pixels comprises foreground points or background points; the cluster classification comprises foreground point clusters or background point clusters;

and M: determining the probability of each pixel belonging to each cluster according to the GMM model of each cluster, and for any pixel, determining the maximum probability value P of the pixel_maxThe corresponding cluster determines the classification of the pixel.

Optionally, determining a segmentation parameter of each pixel on the mask map according to the depth map and the mask map, further includes: determining boundary item segmentation parameters of the pixels according to the depth value difference between the pixels and adjacent pixels;

and for any pixel, accumulating the absolute values of the depth value difference values between the pixel and each adjacent pixel, and then normalizing the accumulated sum to obtain the normalized accumulated sum as a boundary item segmentation parameter of the pixel.

Optionally, constructing an undirected graph, and mapping the partition parameters of each pixel in the mask graph into the undirected graph, including:

constructing an undirected graph, and arranging two suspension points Q outside the plane of the undirected graph₀And Q₁Said suspension point Q₀As a virtual foreground point, the suspension point Q₁Is a virtual background point; establishing mapping points of all pixels on the mask image on the plane of the undirected graph, mapping points of foreground points and the suspension point Q₀A connecting line is established between the mapping point of the background point and the suspension point Q₁Establishing a connection between the two devices;

for any pixel P in the mask image_iThe pixel P_iAs a mapping point P in the undirected graph_i' of the pixel P_iAs a mapping point P in the undirected graph_i' and flying Point Q₀Or Q₁The weight of the line between them.

Optionally, the processing the undirected graph according to a min-max flow algorithm to obtain a finely segmented mask graph includes:

iteratively executing the following steps C and D, stopping the iterative process after the iterative operation reaches a convergence condition, and taking each pixel in the foreground point set Q as a foreground point in a mask image after fine segmentation;

wherein, step C and step D include the following treatment respectively:

c, step C: dividing a part of pixels in an undirected graph into a part of pixels and a floating point Q₀The foreground points of the same kind form a foreground point set Q by pixels divided into the foreground points;

d, step: calculating the weight sum of the foreground point set Q, wherein the weight sum is the weight sum of all foreground points in the foreground point set Q, and all foreground points and suspension points in the foreground point set QQ₀The sum of the weights of the connecting lines;

and the convergence condition is that the sum of the weights of the foreground point set Q is smaller than a threshold value and the change tends to be stable.

Optionally, the original image obtained by photographing the target object is a color image, a black-and-white image, or an infrared image.

Optionally, constructing a geometric figure including the smearing track and expanding the geometric figure to form a mark region, and determining an area including the smearing track as the region of interest includes:

constructing a minimum circumscribed rectangle of the smearing track, and expanding the minimum circumscribed rectangle according to the length L of the smearing track to form a marking area;

applying a geometric template to the smearing track, generating an interested area containing the smearing track according to the geometric template, and if the interested area does not contain the marking area, expanding the interested area to contain the marking area; or performing expansion treatment on the marked region to form a region of interest.

Optionally, the constructing a minimum bounding rectangle of the smearing track, the minimum bounding rectangle being expanded according to the length L of the smearing track to form a mark region, includes:

determining a maximum abscissa value x _ max, a minimum abscissa value x _ min, a maximum ordinate value y _ max and a minimum ordinate value y _ min according to the abscissa and ordinate values of all pixels on the smearing track;

constructing a minimum circumscribed rectangle of the smearing track, wherein coordinates of four vertexes of the minimum circumscribed rectangle are as follows in sequence: (x _ min, y _ max), (x _ min, y _ min), (x _ max, y _ max), (x _ max, y _ min);

and (3) expanding the minimum circumscribed rectangle according to the length L of the smearing track to form a marking area, wherein coordinates of four vertexes of the marking area are (x _ min-a, y _ max + a), (x _ min-a, y _ min-a), (x _ max + a, y _ max + a), (x _ max + a, y _ min-a) in sequence, wherein a is an adjustable coefficient, a is L b, and b is a constant.

Optionally, before generating the input mask map of the image segmentation algorithm, the method further includes:

if the currently detected smearing track is marked for the first time aiming at the original image, taking a marking area determined by the currently detected smearing track as a final marking area, and taking an interested area determined by the currently detected smearing track as a final interested area;

if the currently detected smearing track is not the first mark of the original image, judging whether an intersection exists between a mark area determined by the currently detected smearing track and a mark area determined by the last detected smearing track, if so, merging the mark area determined by the currently detected smearing track and the mark area determined by the last detected smearing track, and taking the merged mark area as a final mark area; if the intersection does not exist, taking the mark area determined by the currently detected smearing track as a final mark area;

if the currently detected smearing track is not the first mark for the original image, judging whether an intersection exists between an interesting region determined by the currently detected smearing track and an interesting region determined by the last detected smearing track, if so, combining the interesting region determined by the currently detected smearing track and the interesting region determined by the last detected smearing track, and taking the combined interesting region as a final interesting region; and if the intersection exists, taking the region of interest determined by the currently detected smearing track as a final region of interest.

Optionally, the image segmentation algorithm is a GrabCut algorithm.

The embodiment of the invention also provides a device for realizing interactive image segmentation, which comprises:

the preprocessing module is used for constructing a geometric figure containing the smearing track and expanding the geometric figure to form a marking area after the smearing track on the original image is detected, and determining an area containing the smearing track as an interested area, wherein the interested area contains the marking area; generating an input mask map for an image segmentation algorithm: all pixels in the marked area are used as foreground points in the mask image, and pixels outside the marked area in the interested area are used as background points in the mask image;

the segmentation parameter calculation module is used for acquiring a depth map containing depth information of a target object, determining segmentation parameters of each pixel on the mask map according to the depth map and the mask map, wherein the segmentation parameters are used for representing the probability that the pixel is judged as a foreground point or a background point and the depth value difference between the pixel and an adjacent pixel;

the mask map adjusting module is used for constructing an undirected graph, mapping the segmentation parameters of each pixel in the mask map into the undirected graph, and processing the undirected graph according to a minimum segmentation-maximum flow algorithm to obtain a finely segmented mask map;

and the output module is used for segmenting an image corresponding to the foreground point in the finely segmented mask image from an original image obtained by shooting the target object.

Optionally, the segmentation parameter calculation module is configured to determine the segmentation parameters of each pixel on the mask map according to the depth map and the mask map in the following manner: performing Gaussian Mixture Model (GMM) calculation according to an EM (effective electromagnetic) method, wherein the EM method comprises an E step and an M step; e and M steps of iterative operation, stopping the iterative process after the iterative operation reaches a convergence condition; determining the classification of the pixel obtained by the last M steps as the classification of the pixel, and determining the maximum probability value P of the pixel belonging to a cluster obtained by the last M steps_maxDetermining a region item segmentation parameter of the pixel, wherein the region item segmentation parameter is the probability that the pixel is judged to be a foreground point or a background point;

Optionally, the segmentation parameter calculation module is further configured to determine the segmentation parameters of each pixel on the mask map according to the depth map and the mask map in the following manner: determining boundary item segmentation parameters of the pixels according to the depth value difference between the pixels and adjacent pixels;

Optionally, the mask map adjusting module is configured to construct an undirected graph and map the partition parameters of each pixel in the mask map into the undirected graph in the following manner:

Optionally, the mask map adjusting module is configured to process the undirected graph according to a min-max flow algorithm in the following manner to obtain a finely segmented mask map:

wherein, step C and step D include the following treatment respectively:

d, step: calculating the weight sum of the foreground point set Q, wherein the weight sum is the weight sum of all foreground points in the foreground point set Q, and all foreground points and suspension points Q in the foreground point set Q are added₀The sum of the weights of the connecting lines;

Optionally, the preprocessing module is configured to construct a geometric figure including the smearing track and expand the geometric figure to form a mark region, and determine an area including the smearing track as the region of interest:

Optionally, the preprocessing module is configured to construct a minimum bounding rectangle of the smearing track and expand the minimum bounding rectangle to form a mark region according to the length L of the smearing track by:

Optionally, the preprocessing module is further configured to perform an adjustment of the marking region and the region of interest before generating the input mask map of the image segmentation algorithm:

Optionally, the image segmentation algorithm is a GrabCut algorithm.

The embodiment of the invention also provides a terminal which comprises the device for realizing the interactive image segmentation.

The method, the device and the terminal for realizing interactive image segmentation are provided, a geometric figure containing a smearing track is constructed and expanded to form a marking area, and an interested area containing the marking area is determined; generating an input mask map for an image segmentation algorithm: all pixels in the marked area are used as foreground points in the mask image, and pixels outside the marked area in the interested area are used as background points in the mask image; determining segmentation parameters of each pixel according to a depth map and a mask map, wherein the segmentation parameters are used for representing the probability of judging the pixel as a foreground point or a background point and the depth value difference between the pixel and an adjacent pixel, constructing an undirected graph, mapping the segmentation parameters of each pixel in the mask map into the undirected graph, running a minimum segmentation Mincut-maximum flow MaxFlow algorithm according to the undirected graph to obtain a finely segmented mask map, and segmenting an image corresponding to a foreground point in the finely segmented mask map from an original image obtained by shooting the target object. The technical scheme can expand the number of foreground points marked by the image segmentation algorithm through image preprocessing, reduce the number of background points marked by the image segmentation algorithm, and further shorten the running time of the image segmentation algorithm.

Drawings

Fig. 1 is a schematic diagram of a hardware structure of an optional mobile terminal for implementing various embodiments of the present invention;

FIG. 2 is a diagram of a wireless communication system for the mobile terminal shown in FIG. 1;

FIG. 3 is a flowchart of a method for implementing interactive image segmentation according to embodiment 1 of the present invention;

fig. 4 is a schematic diagram of an apparatus for implementing interactive image segmentation according to embodiment 2 of the present invention;

FIG. 5-a is a schematic diagram of an original image and a smear track of a user in application example 1 of the present invention;

FIG. 5-b is a schematic diagram of a marking region and a region of interest generated by a smear track in application example 1 of the present invention;

FIG. 5-c is a schematic view of a mask map generated from a mark region and a region of interest in application example 1 of the present invention;

fig. 5-d is a schematic view of a depth map of application example 1 of the present invention;

fig. 5-e is a schematic diagram of an undirected graph of application example 1 of the present invention;

FIG. 5 f is a schematic view of a mask pattern after fine division in application example 1 of the present invention;

FIG. 5 g is a schematic view of a target object divided in application example 1 of the present invention;

FIG. 6-a-1 is a schematic diagram of an original image and a first smearing trajectory by a user in application example 2 of the present invention;

6-a-2 are schematic diagrams of an original image and a second smearing trajectory by a user in application example 2 of the present invention;

FIG. 6-b-1 is a schematic view of a mark region and a region of interest generated by a first smearing trajectory in application example 2 of the present invention;

6-b-2 is a schematic diagram of a marking region and a region of interest generated by a second smearing track in application example 2 of the present invention;

FIG. 6-c-1 is a schematic view of a mask pattern generated by the first smear in application example 2 of the present invention;

fig. 6-c-2 is a schematic diagram of a mask pattern generated by the second smearing in application example 2 of the present invention.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

The technical solution of the present invention will be described in more detail with reference to the accompanying drawings and examples.

A mobile terminal implementing various embodiments of the present application will now be described with reference to the accompanying drawings. In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "module" and "component" may be used in a mixture.

The mobile terminal may be implemented in various forms. For example, the terminal described in the present invention may include a mobile terminal such as a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. In the following, it is assumed that the terminal is a mobile terminal. However, it will be understood by those skilled in the art that the configuration according to the embodiment of the present invention can be applied to a fixed type terminal in addition to elements particularly used for moving purposes.

Fig. 1 is a schematic diagram of a hardware structure of an optional mobile terminal for implementing various embodiments of the present application.

The mobile terminal 100 may include a wireless communication unit 110, an a/V (audio/video) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, an interface unit 170, a controller 180, and a power supply unit 190, etc.

Fig. 1 illustrates the mobile terminal 100 having various components, but it is to be understood that not all illustrated components are required to be implemented. More or fewer components may alternatively be implemented. The elements of the mobile terminal 100 will be described in detail below.

The wireless communication unit 110 may generally include one or more components that allow radio communication between the mobile terminal 100 and a wireless communication system or network. For example, the wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless internet module 113, a short-range communication module 114, and a location information module 115.

The broadcast receiving module 111 receives a broadcast signal and/or broadcast-related information from an external broadcast management server via a broadcast channel, the broadcast channel may include a satellite channel and/or a terrestrial channel, the broadcast management server may be a server that generates and transmits a broadcast signal and/or broadcast-related information or a server that receives a previously generated broadcast signal and/or broadcast-related information and transmits it to a terminal, the broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, etc. also, the broadcast signal may further include a broadcast signal combined with the TV or radio broadcast signal, the broadcast-related information may also be provided via a mobile communication network, and in this case, the broadcast-related information may be received by the mobile communication module 112, the broadcast signal may exist in various forms, for example, it may exist in the form of an Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB), a digital video broadcasting handheld (DVB-H) Electronic Service Guide (ESG), etc., the broadcast-receiving module 111 may receive a signal broadcast using various types of broadcast systems, in particular, the broadcast receiving module 111 may receive a broadcast signal via a media storage medium such as a multimedia broadcast-terrestrial broadcast-storage medium (digital broadcast-terrestrial broadcast-storage medium, DVB-H, etc., the foregoing digital broadcast-DVB-storage medium, DVB-storage medium, and digital broadcast storage medium, DVB-storage medium, and digital broadcast storage medium suitable for receiving system storage medium, DMB-DVB-.

The mobile communication module 112 transmits and/or receives radio signals to and/or from at least one of a base station (e.g., access point, node B, etc.), an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data transmitted and/or received according to text and/or multimedia messages.

The wireless internet access technology to which the module refers may include W L AN (wireless L AN) (Wi-Fi), Wibro (wireless broadband), Wimax (worldwide interoperability for microwave Access), HSDPA (high speed Downlink packet Access), and the like.

The short-range communication module 114 is a module for supporting short-range communication. Some examples of short-range communication technologies include bluetooth (TM), Radio Frequency Identification (RFID), infrared data association (IrDA), Ultra Wideband (UWB), zigbee (TM), and the like.

The location information module 115 is a module for checking or acquiring location information of the mobile terminal. A typical example of the location information module 115 is a GPS (global positioning system). According to the current technology, the GPS calculates distance information and accurate time information from three or more satellites and applies triangulation to the calculated information, thereby accurately calculating three-dimensional current location information according to longitude, latitude, and altitude. Currently, a method for calculating position and time information uses three satellites and corrects an error of the calculated position and time information by using another satellite. In addition, the GPS can calculate speed information by continuously calculating current position information in real time.

The a/V input unit 120 is used to receive an audio or video signal. The a/V input unit 120 may include a camera 121 and a microphone 122, and the camera 121 processes image data of still pictures or video obtained by an image capturing apparatus in a video capturing mode or an image capturing mode. The processed image frames may be displayed on the display unit 151. The image frames processed by the cameras 121 may be stored in the memory 160 (or other storage medium) or transmitted via the wireless communication unit 110, and two or more cameras 121 may be provided according to the construction of the mobile terminal 100. The microphone 122 may receive sounds (audio data) via the microphone 122 in a phone call mode, a recording mode, a voice recognition mode, or the like, and is capable of processing such sounds into audio data. The processed audio (voice) data may be converted into a format output transmittable to a mobile communication base station via the mobile communication module 112 in case of a phone call mode. The microphone 122 may implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated in the course of receiving and transmitting audio signals.

The user input unit 130 may generate key input data to control various operations of the mobile terminal 100 according to a command input by a user. The user input unit 130 allows a user to input various types of information, and may include a keyboard, dome sheet, touch pad (e.g., a touch-sensitive member that detects changes in resistance, pressure, capacitance, and the like due to being touched), scroll wheel, joystick, and the like. In particular, when the touch pad is superimposed on the display unit 151 in the form of a layer, a touch screen may be formed.

The sensing unit 140 detects a current state of the mobile terminal 100 (e.g., an open or closed state of the mobile terminal 100), a position of the mobile terminal 100, presence or absence of contact (i.e., touch input) by a user with the mobile terminal 100, an orientation of the mobile terminal 100, acceleration or deceleration movement and direction of the mobile terminal 100, and the like, and generates a command or signal for controlling an operation of the mobile terminal 100. For example, when the mobile terminal 100 is implemented as a slide-type mobile phone, the sensing unit 140 may sense whether the slide-type phone is opened or closed. In addition, the sensing unit 140 can detect whether the power supply unit 190 supplies power or whether the interface unit 170 is coupled with an external device. The sensing unit 140 may include a proximity sensor 141.

The interface unit 170 serves as an interface through which at least one external device is connected to the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The identification module may store various information for authenticating a user using the mobile terminal 100 and may include a User Identity Module (UIM), a Subscriber Identity Module (SIM), a Universal Subscriber Identity Module (USIM), and the like. In addition, a device having an identification module (hereinafter, referred to as an "identification device") may take the form of a smart card, and thus, the identification device may be connected with the mobile terminal 100 via a port or other connection means. The interface unit 170 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and the external device.

In addition, when the mobile terminal 100 is connected with an external cradle, the interface unit 170 may serve as a path through which power is supplied from the cradle to the mobile terminal 100 or may serve as a path through which various command signals input from the cradle are transmitted to the mobile terminal 100. Various command signals or power input from the cradle may be used as a signal for identifying whether the mobile terminal 100 is accurately mounted on the cradle. The output unit 150 is configured to provide output signals (e.g., audio signals, video signals, alarm signals, vibration signals, etc.) in a visual, audio, and/or tactile manner. The output unit 150 may include a display unit 151, an audio output module 152, an alarm unit 153, and the like.

The display unit 151 may display information processed in the mobile terminal 100. For example, when the mobile terminal 100 is in a phone call mode, the display unit 151 may display a User Interface (UI) or a Graphical User Interface (GUI) related to a call or other communication (e.g., text messaging, multimedia file downloading, etc.). When the mobile terminal 100 is in a video call mode or an image capturing mode, the display unit 151 may display a captured image and/or a received image, a UI or GUI showing a video or an image and related functions, and the like.

Meanwhile, when the display unit 151 and the touch pad are stacked on each other in the form of a layer TO form a touch screen, the display unit 151 may be used as an input device and an output device, the display unit 151 may include at least one of a liquid crystal display (L CD), a thin film transistor L CD (TFT-L CD), an organic light emitting diode (O L ED) display, a flexible display, a three-dimensional (3D) display, and the like, some of these displays may be configured TO be transparent TO allow a user TO view from the outside, which may be referred TO as a transparent display, a typical transparent display may be, for example, a TO L ED (transparent organic light emitting diode) display, and the like, the mobile terminal 100 may include two or more display units (or other display devices) according TO a specific intended embodiment, for example, the mobile terminal 100 may include an external display unit (not shown) and an internal display unit (not shown), and the touch screen may be used TO detect a touch input pressure and a touch input position and a touch input area.

The audio output module 152 may convert audio data received by the wireless communication unit 110 or stored in the memory 160 into an audio signal and output as sound when the mobile terminal 100 is in a call signal reception mode, a call mode, a recording mode, a voice recognition mode, a broadcast reception mode, or the like. Also, the audio output module 152 may provide audio output related to a specific function performed by the mobile terminal 100 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output module 152 may include a speaker, a buzzer, and the like.

The alarm unit 153 may provide an output to notify the mobile terminal 100 of the occurrence of an event. Typical events may include call reception, message reception, key signal input, touch input, and the like. In addition to audio or video output, the alarm unit 153 may provide output in different ways to notify the occurrence of an event. For example, the alarm unit 153 may provide an output in the form of vibration, and when a call, a message, or some other incoming communication (communicating communication) is received, the alarm unit 153 may provide a tactile output (i.e., vibration) to inform the user thereof. By providing such a tactile output, the user can recognize the occurrence of various events even when the user's mobile phone is in the user's pocket. The alarm unit 153 may also provide an output notifying the occurrence of an event via the display unit 151 or the audio output module 152.

The memory 160 may store software programs and the like for processing and controlling operations performed by the controller 180, or may temporarily store data (e.g., a phonebook, messages, still images, videos, and the like) that has been or will be output. Also, the memory 160 may store data regarding various ways of vibration and audio signals output when a touch is applied to the touch screen.

The memory 160 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. Also, the mobile terminal 100 may cooperate with a network storage device that performs a storage function of the memory 160 through a network connection.

The controller 180 generally controls the overall operation of the mobile terminal. For example, the controller 180 performs control and processing related to voice calls, data communications, video calls, and the like. In addition, the controller 180 may include a multimedia module 181 for reproducing (or playing back) multimedia data, and the multimedia module 181 may be constructed within the controller 180 or may be constructed separately from the controller 180. The controller 180 may perform a pattern recognition process to recognize a handwriting input or a picture drawing input performed on the touch screen as a character or an image.

The power supply unit 190 receives external power or internal power and provides appropriate power required to operate various elements and components under the control of the controller 180.

For a hardware implementation, the embodiments described herein may be implemented using at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a programmable logic device (P L D), a Field Programmable Gate Array (FPGA), a processor, a controller, a microcontroller, a microprocessor, an electronic unit designed to perform the functions described herein, such an implementation may be implemented in the controller 180 in some cases.

Up to this point, the mobile terminal 100 has been described in terms of its functionality. In addition, the mobile terminal 100 in the embodiment of the present invention may be a mobile terminal such as a folder type, a bar type, a swing type, a slide type, and other various types, and is not limited herein.

The mobile terminal 100 as shown in fig. 1 may be configured to operate with communication systems such as wired and wireless communication systems and satellite-based communication systems that transmit data via frames or packets.

A communication system in which a mobile terminal according to the present invention is operable will now be described with reference to fig. 2.

For example, the air interfaces used by communication systems include, for example, Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), and Universal Mobile Telecommunications System (UMTS) (particularly Long term evolution (L TE)), Global System for Mobile communications (GSM), and so forth.

Referring to fig. 2, a CDMA wireless communication system may include a plurality of mobile terminals 100, a plurality of Base Stations (BSs) 270, Base Station Controllers (BSCs) 275, and Mobile Switching Centers (MSCs) 280, the MSCs 280 are structured to interface with Public Switched Telephone Networks (PSTNs) 290, the MSCs 280 are also structured to interface with the BSCs 275, which may be coupled to the base stations 270 via backhaul lines, backhaul lines may be structured according to any of several known interfaces, which may include, for example, european standard high capacity digital lines/U.S. standard high capacity digital lines (E1/T1), Asynchronous Transfer Mode (ATM), network protocol (IP), point-to-point protocol (PPP), frame relay, high rate digital subscriber lines (HDS L), asymmetric digital subscriber lines (ADS L), or various types of digital subscriber lines (xDS L), it will be appreciated that a system as shown in fig. 2 may include a plurality of BSCs 275.

Each BS 270 may serve one or more sectors (or regions), each sector covered by a multi-directional antenna or an antenna pointing in a particular direction being radially distant from the BS 270. Alternatively, each partition may be covered by two or more antennas for diversity reception. Each BS 270 may be configured to support multiple frequency allocations, with each frequency allocation having a particular frequency spectrum (e.g., 1.25MHz, 5MHz, etc.).

The intersection of partitions with frequency allocations may be referred to as a CDMA channel. The BS 270 may also be referred to as a Base Transceiver Subsystem (BTS) or other equivalent terminology. In such a case, the term "base station" may be used to generically refer to a single BSC275 and at least one BS 270. The base stations may also be referred to as "cells". Alternatively, each partition of a particular BS 270 may be referred to as a plurality of cell sites.

As shown in fig. 2, a Broadcast Transmitter (BT)295 transmits a broadcast signal to the mobile terminal 100 operating within the system. A broadcast receiving module 111 as shown in fig. 1 is provided at the mobile terminal 100 to receive a broadcast signal transmitted by the BT 295. In fig. 2, several Global Positioning System (GPS) satellites 300 are shown. The satellite 300 assists in locating at least one of the plurality of mobile terminals 100.

In fig. 2, a plurality of satellites 300 are depicted, but it is understood that useful positioning information may be obtained with any number of satellites. The location information module 115 (e.g., GPS) as shown in fig. 1 is generally configured to cooperate with the satellites 300 to obtain desired positioning information. Other techniques that can track the location of the mobile terminal may be used instead of or in addition to GPS tracking techniques. In addition, at least one GPS satellite 300 may selectively or additionally process satellite DMB transmission.

As a typical operation of the wireless communication system, the BS 270 receives reverse link signals from various mobile terminals 100. The mobile terminal 100 is generally engaged in conversations, messaging, and other types of communications. Each reverse link signal received by a particular base station is processed within a particular BS 270. The obtained data is forwarded to the associated BSC 275. The BSC provides call resource allocation and mobility management functions including coordination of soft handoff procedures between BSs 270. The BSCs 275 also route the received data to the MSC 280, which provides additional routing services for interfacing with the PSTN 290. Similarly, the PSTN290 interfaces with the MSC 280, the MSC interfaces with the BSCs 275, and the BSCs 275 accordingly control the BS 270 to transmit forward link signals to the mobile terminal 100.

Based on the above mobile terminal hardware structure and communication system, various embodiments of the method of the present application are proposed.

As shown in fig. 3, an embodiment of the present invention provides a method for implementing interactive image segmentation, including:

s310, after detecting the smearing track on the original image, constructing a geometric figure containing the smearing track, expanding the geometric figure to form a marking area, and determining an area containing the smearing track as an interested area, wherein the interested area contains the marking area; generating an input mask map for an image segmentation algorithm: all pixels in the marked area are used as foreground points in the mask image, and pixels outside the marked area in the interested area are used as background points in the mask image;

s320, obtaining a depth map containing depth information of the target object, and determining segmentation parameters of each pixel on the mask map according to the depth map and the mask map, wherein the segmentation parameters are used for representing the probability that the pixel is judged as a foreground point or a background point and the depth value difference between the pixel and an adjacent pixel;

s330, constructing an undirected graph, mapping the segmentation parameters of each pixel in the mask graph into the undirected graph, and processing the undirected graph according to a minimum segmentation-maximum flow algorithm to obtain a finely segmented mask graph;

s340, segmenting an image corresponding to a foreground point in the finely segmented mask image from an original image obtained by shooting the target object;

the method may further comprise the following features:

the smearing is a mode of marking the target object, and the smearing is to mark the inner area of the target object;

the mask image is a mark image generated after foreground and background distinguishing is carried out on part or all pixels of one image, and each pixel on the mask image is marked as a foreground point or a background point.

In this embodiment, the image segmentation algorithm is the GrabCut algorithm.

In one embodiment, the constructing and expanding the geometric figure containing the smearing track to form a mark area, and the determining an area containing the smearing track as the region of interest includes:

applying a geometric template for the smearing track, generating a Region of Interest (ROI) containing the smearing track according to the geometric template, and if the Region of Interest does not contain the mark Region, expanding the Region of Interest to contain the mark Region; or performing expansion treatment on the marked region to form a region of interest.

Wherein the geometric figure comprises: rectangular, polygonal, elliptical, etc.

Wherein the geometric form comprises: a rectangular template, a polygonal template, or an elliptical template;

wherein the constructing a minimum bounding rectangle of the smearing track, the expanding the minimum bounding rectangle according to the length L of the smearing track to form a marking region, comprises:

the minimum circumscribed rectangle is expanded according to the length L of the smearing track to form a marking area, and coordinates of four vertexes of the marking area are sequentially (x _ min-a, y _ max + a), (x _ min-a, y _ min-a), (x _ max + a, y _ max + a), (x _ max + a, y _ min-a), wherein a is an adjustable coefficient, a is L b, b is a constant, and b can be an empirical value.

In this embodiment, determining the segmentation parameters of each pixel on the mask map according to the depth map and the mask map includes:

performing Gaussian Mixture Model (GMM) calculation according to an EM (effective electromagnetic) method, wherein the EM method comprises an E step and an M step; e and M steps of iterative operation, stopping the iterative process after the iterative operation reaches a convergence condition; determining the classification of the pixel obtained by the last M steps as the classification of the pixel, and determining the maximum probability value P of the pixel belonging to a cluster obtained by the last M steps_maxDetermining a region item segmentation parameter of the pixel, wherein the region item segmentation parameter is the probability that the pixel is judged to be a foreground point or a background point;

and M: determining the probability of each pixel belonging to each cluster according to the GMM model of each cluster, and for any pixel, determining the maximum probability value P of the pixel_maxThe corresponding cluster determines the classification of the pixel;

optionally, the convergence condition of the EM method may be: stopping the iteration process when the number of times of iteration operation reaches a threshold value;

in this embodiment, determining the segmentation parameters of each pixel on the mask map according to the depth map and the mask map further includes: determining boundary item segmentation parameters of the pixels according to the depth value difference between the pixels and adjacent pixels;

accumulating the absolute values of the depth value difference values between any one pixel and each adjacent pixel, and then normalizing the accumulated sum to obtain the normalized accumulated sum as a boundary item segmentation parameter of the pixel;

alternatively, each pixel adjacent to a pixel may be 8 pixels of the periphery of the pixel;

in this embodiment, constructing an undirected graph, and mapping the partition parameters of each pixel in the mask graph into the undirected graph includes:

for any pixel P in the mask image_iThe pixel P_iAs a mapping point P in the undirected graph_i' of the pixel P_iAs a mapping point P in the undirected graph_i' and flying Point Q₀Or Q₁The weight of the connecting line between the two;

in this embodiment, the processing the undirected graph according to the min-max flow algorithm to obtain a finely divided mask graph includes:

wherein, step C and step D include the following treatment respectively:

d, step: calculating the weight value of the foreground point set QThe sum of the weights is the sum of the weights of all the foreground points in the foreground point set Q, and all the foreground points and the suspension points Q in the foreground point set Q₀The sum of the weights of the connecting lines;

the convergence condition is that the sum of the weights of the foreground point set Q is smaller than a threshold value and the change tends to be stable;

alternatively, the original image obtained by photographing the target object may be a color image, a black-and-white image, or an infrared image.

In one embodiment, prior to generating the input mask map for the image segmentation algorithm, the method further comprises:

Besides the first smearing, the mark area and the interesting area corresponding to each subsequent smearing are compared with the mark area and the interesting area determined by the previous smearing track, so that foreground points and background points in the input mask image are more accurate, and the quality of the segmented image is improved.

In the related technology, foreground points in an input mask image of an image segmentation algorithm are manually marked by a user, other pixels except the foreground points on an original image are marked as background points, and the problems that iteration times of distinguishing the foreground points and the background points by the image segmentation algorithm are increased due to the fact that the foreground points are marked less and the size of the input mask image is large, and running time of the algorithm is long exist. After the method provided by the embodiment of the invention is adopted, the mark area is generated by constructing the geometric image, and the pixels in the mark area are marked as foreground points, so that the number of foreground points marked in an input mask image of the image segmentation algorithm is automatically expanded, the number of background points marked by the image segmentation algorithm can be reduced by generating the region of interest to replace the whole original image, the iteration frequency of distinguishing the foreground points and the background points by the image segmentation algorithm can be reduced, and the running time of the image segmentation algorithm is remarkably reduced. On the other hand, the technical scheme of the embodiment of the invention utilizes the depth map to calculate the segmentation parameters, and can improve the image segmentation effect when the color features of the target object to be extracted are not obvious.

As shown in fig. 4, an embodiment of the present invention provides an apparatus for implementing interactive image segmentation, including:

the preprocessing module 401 is configured to, after detecting a smearing track on an original image, construct a geometric figure including the smearing track and expand the geometric figure to form a marking region, and determine an area including the smearing track as an area of interest, where the area of interest includes the marking region; generating an input mask map for an image segmentation algorithm: all pixels in the marked area are used as foreground points in the mask image, and pixels outside the marked area in the interested area are used as background points in the mask image;

a segmentation parameter calculation module 402, configured to obtain a depth map including depth information of a target object, and determine a segmentation parameter of each pixel on the mask map according to the depth map and the mask map, where the segmentation parameter is used to indicate a probability that a pixel is determined as a foreground point or a background point and a depth value difference between the pixel and an adjacent pixel;

the mask map adjusting module 403 is configured to construct an undirected graph, map the segmentation parameters of each pixel in the mask map into the undirected graph, and process the undirected graph according to a min-max flow algorithm to obtain a finely segmented mask map;

an output module 404, configured to segment an image corresponding to a foreground point in the finely segmented mask map from an original image obtained by capturing the target object;

the apparatus may also include the following features:

In this embodiment, the image segmentation algorithm is the GrabCut algorithm.

In one embodiment, the preprocessing module is configured to construct a geometric figure containing the smearing track and expand the geometric figure to form a mark area, and determine an area containing the smearing track as the region of interest:

In this embodiment, the segmentation parameter calculation module is configured to determine the segmentation parameter of each pixel on the mask map according to the depth map and the mask map in the following manner:

in this embodiment, the segmentation parameter calculation module is further configured to determine the segmentation parameters of each pixel on the mask map according to the depth map and the mask map in the following manner: determining boundary item segmentation parameters of the pixels according to the depth value difference between the pixels and adjacent pixels;

alternatively, each pixel adjacent to a pixel may be 8 pixels of the periphery of the pixel.

In this embodiment, the mask map adjusting module is configured to construct an undirected graph and map the partition parameters of each pixel in the mask map into the undirected graph in the following manner:

constructing an undirected graph, and arranging two suspension points Q outside the plane of the undirected graph₀And Q₁Said suspension point Q₀As a virtual foreground point, the suspension point Q₁Is a virtual background point; establishing mapping points of all pixels on the mask image on the plane of the undirected graph, mapping points of foreground points and the suspension point Q₀Establishing a connection line between the mapping point of the background point and the mapping point of the background pointThe suspension point Q₁Establishing a connection between the two devices;

In this embodiment, the mask map adjusting module is configured to process the undirected graph according to a min-max flow algorithm in the following manner to obtain a finely divided mask map:

wherein, step C and step D include the following treatment respectively:

In one embodiment, the preprocessing module is further configured to perform the adjustment of the marker region and the region of interest before generating the input mask map for the image segmentation algorithm:

According to the method provided by the embodiment of the invention, the mark area is generated by constructing the geometric image, the pixels in the mark area are marked as foreground points, so that the number of foreground points marked in an input mask image of the image segmentation algorithm is automatically expanded, the number of background points marked by the image segmentation algorithm can be reduced by generating the region of interest to replace the whole original image, the iteration frequency of distinguishing the foreground points and the background points by the image segmentation algorithm can be reduced, and the running time of the image segmentation algorithm is remarkably reduced. On the other hand, the technical scheme of the embodiment of the invention utilizes the depth map to calculate the segmentation parameters, and can improve the image segmentation effect when the color features of the target object to be extracted are not obvious.

Example 3

Application example 1

The method for preprocessing the image by using the preprocessing method comprises the following steps:

step S501, detecting that a user selects to mark a target object in a smearing mode;

for example, two keys for marking are provided on the interface, one is "smearing", and the other is "outlining", and if the user clicks the "smearing" key, the smearing trajectory is preprocessed.

Wherein, smearing and outlining are two different modes of marking the target object;

generally, smearing is to mark an inner region of the target object, and delineating is to mark along an outer contour of the target object;

step S502, detecting that a user paints on an original image;

for example, as shown in fig. 5-a, the user has painted on an original image, which is a color map, and the target object is a "stapler";

step S503, constructing a minimum circumscribed rectangle of the smearing track, expanding the minimum circumscribed rectangle according to the length L of the smearing track to form a marking area, applying a rectangular template for the smearing track, and generating an interested area containing the smearing track according to the rectangular template;

expanding the minimum circumscribed rectangle according to the length L of the smearing track to form a marking area, wherein coordinates of four vertexes of the marking area are sequentially (x _ min-a, y _ max + a), (x _ min-a, y _ min-a), (x _ max + a, y _ max + a), (x _ max + a, y _ min-a), wherein a is an adjustable coefficient, a is L b, and b is a constant;

for example, as shown in fig. 5-b, the marking area corresponding to the smearing track is a first rectangular area containing the smearing track, the region of interest is a second rectangular area containing the marking area, and the border of the region of interest is indicated by a dashed line.

Step S504, generating an input mask map of an image segmentation algorithm (GrabCut algorithm): all pixels in the mark area are used as foreground points in the mask image, and pixels outside the mark area in the interested area are used as background points in the mask image.

For example, as shown in fig. 5-c, the rectangular dark region containing the smear track is a mark region, which is a foreground point block in the mask map (input mask map); the edge of the region of interest is represented by a dashed box, inside which the part excluding the marked area (foreground dot block) is the background dot block in the mask map.

Step S505, acquiring a depth map containing depth information of the target object;

as shown in fig. 5-d, the depth map is a map containing depth information, corresponding to the size of the color map; in the depth map, the darker part is shot farther, and the lighter part is shot closer.

Step S506, determining segmentation parameters of each pixel on the mask map according to the depth map and the mask map, wherein the segmentation parameters are used for representing the probability that the pixel is judged as a foreground point or a background point and the depth value difference between the pixel and an adjacent pixel;

for any pixel on the mask image, the segmentation parameters of the pixel comprise region term segmentation parameters and boundary term segmentation parameters; the region item segmentation parameter of the pixel refers to the probability that the pixel is judged as a foreground point or a background point; the boundary item segmentation parameter of the pixel refers to the depth value difference of the pixel and the adjacent pixel;

wherein, the area item segmentation parameter of each pixel on the mask map is determined by adopting the following method:

performing Gaussian Mixture Model (GMM) calculation according to an EM (effective electromagnetic) method, wherein the EM method comprises an E step and an M step; e and M iterative operation steps, stopping the iterative process after the iterative operation times reach the designated times; determining the classification of the pixel obtained by the last M steps as the classification of the pixel, and determining the maximum probability value P of the pixel belonging to a cluster obtained by the last M steps_maxDetermining a region item segmentation parameter of the pixel, wherein the region item segmentation parameter is the probability that the pixel is judged to be a foreground point or a background point;

wherein the boundary term segmentation parameters of each pixel on the mask map are determined in the following way: accumulating the absolute values of the depth value difference values between any pixel and each pixel in the adjacent 8-neighborhood, and then normalizing the accumulated sum to obtain the normalized accumulated sum as a boundary item segmentation parameter of the pixel;

step S507, constructing an undirected graph, mapping the segmentation parameters of each pixel in the mask graph into the undirected graph, and processing the undirected graph according to a minimum segmentation-maximum flow algorithm to obtain a finely segmented mask graph;

wherein, the undirected graph is shown in fig. 5-e, and two suspension points Q are arranged outside the plane of the undirected graph₀And Q₁Said suspension point Q₀As a virtual foreground point, the suspension point Q₁Is a virtual background point; establishing mapping points of all pixels on the mask image on the plane of the undirected graph, mapping points of foreground points and the suspension point Q₀A connecting line is established between the mapping point of the background point and the suspension point Q₁Establishing a connection between the two devices;

And running a minimum cut Mincut-maximum flow MaxFlow algorithm according to the undirected graph in the following mode to obtain a finely divided mask graph:

wherein, step C and step D include the following treatment respectively:

wherein, the mask image after the fine segmentation is as shown in fig. 5-f, the irregular dark region containing the smear track and having the contour of the target image (stapler) is a mark region, and the mark region is a foreground point block in the mask image after the fine segmentation; the part of the region of interest excluding the marked area (foreground point block) is the background point block in the mask map. The boundary between the foreground block of dots and the background block of dots is finer in the finely divided mask pattern than in the original mask pattern.

Step S508, the image corresponding to the foreground point in the mask image after the fine segmentation is segmented from the original image obtained by shooting the target object;

wherein the target object is segmented from the original color map based on the finely segmented mask map, the segmented "stapler" image being shown in fig. 5-g.

According to the method of the application example, all pixels in the mark area are marked as foreground points by determining the first rectangular adjacent area into which the smearing track is expanded as the mark area, the number of foreground points marked in an input mask image of the image segmentation algorithm can be automatically expanded, the number of background points marked by the image segmentation algorithm can be reduced by taking the second rectangular area containing the smearing track as the area of interest to replace the whole original image, the iteration frequency of distinguishing the foreground points and the background points by the image segmentation algorithm can be reduced, and the running time of the image segmentation algorithm is remarkably reduced. On the other hand, the technical scheme of the embodiment of the invention utilizes the depth map to calculate the segmentation parameters, and can improve the image segmentation effect when the color features of the target object to be extracted are not obvious.

Application example 2

The method comprises the following steps that a user paints a target object interested by the user on an original image, the image is preprocessed by adopting a preprocessing method of the text, and multiple input (multiple painting) can be accepted, and the method comprises the following steps:

step S601, detecting that a user selects to mark a target object in a smearing mode;

Step S602, detecting that a user paints on an original image;

for example, as shown in FIG. 6-a-1, the user has first painted the original image, and the target object is a "stapler";

for example, as shown in fig. 6-a-2, the user has applied a second smear on the original image, and the target object is a "stapler";

step S603, constructing a minimum circumscribed rectangle of the smearing track, expanding the minimum circumscribed rectangle according to the length L of the smearing track to form a marking area, applying a rectangular template for the smearing track, and generating an interested area containing the smearing track according to the rectangular template;

for example, as shown in fig. 6-b-1, the mark area corresponding to the first smearing track is a first rectangular area containing the first smearing track, the region of interest is a second rectangular area containing the mark area, and the border of the region of interest is indicated by a dashed line;

for example, as shown in fig. 6-b-2, the marking area corresponding to the second smearing track is a first rectangular area containing the second smearing track, the region of interest is a second rectangular area containing the marking area, and a frame of the region of interest is indicated by a dashed line;

step S604, generating an input mask map of the image segmentation algorithm, including:

if the currently detected smearing track is not the first mark for the original image, judging whether an intersection exists between an interesting region determined by the currently detected smearing track and an interesting region determined by the last detected smearing track, if so, combining the interesting region determined by the currently detected smearing track and the interesting region determined by the last detected smearing track, and taking the combined interesting region as a final interesting region; if the intersection does not exist, taking the region of interest determined by the currently detected smearing track as a final region of interest;

all pixels in the marked area are used as foreground points in the mask image, and pixels outside the marked area in the interested area are used as background points in the mask image;

for example, as shown in FIG. 6-c-1, the rectangular dark region containing the first smearing track is a marker region, which is a block of foreground points in the mask map (input mask map); the edge of the region of interest is represented by a dashed box, inside which the part excluding the marked area (foreground dot block) is the background dot block in the mask map.

For example, as shown in fig. 6-c-2, if there is an intersection between the mark region corresponding to the first smearing track and the mark region corresponding to the second smearing track, the final mark region is a union of the two mark regions; and if an intersection exists between the region of interest corresponding to the first smearing track and the region of interest corresponding to the second smearing track, the final region of interest is the union of the two regions of interest. The marked region is a block of foreground points in a mask map (input mask map); the edge of the region of interest is represented by a dashed box, inside which the part excluding the marked area (foreground dot block) is the background dot block in the mask map.

Step S605, acquiring a depth map containing depth information of the target object;

step S606, determining segmentation parameters of each pixel on the mask map according to the depth map and the mask map, wherein the segmentation parameters are used for representing the probability that the pixel is judged as a foreground point or a background point and the depth value difference between the pixel and an adjacent pixel;

step S607, constructing an undirected graph, mapping the segmentation parameters of each pixel in the mask graph into the undirected graph, and operating a minimum-cut Mincut-maximum-flow MaxFlow algorithm according to the undirected graph to obtain a finely segmented mask graph;

step S608, segmenting an image corresponding to a foreground point in the mask image after the fine segmentation from an original image obtained by shooting the target object;

wherein an image of the "stapler" is segmented from the original color map on the basis of the finely segmented mask map.

Besides the first smearing, the mark area and the interesting area corresponding to each subsequent smearing are compared with the mark area and the interesting area determined by the previous smearing track, so that foreground points and background points in the input mask image are more accurate, and the quality of the segmented image is improved;

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. A method of implementing interactive image segmentation, comprising:

segmenting an image corresponding to a foreground point in the finely segmented mask image from an original image obtained by photographing the target object,

determining segmentation parameters of each pixel on the mask map according to the depth map and the mask map, wherein the segmentation parameters comprise: performing Gaussian Mixture Model (GMM) calculation according to an EM (effective electromagnetic) method, wherein the EM method comprises an E step and an M step; e and M steps of iterative operation, stopping the iterative process after the iterative operation reaches a convergence condition; determining the classification of the pixel obtained by the last M steps as the classification of the pixel, and determining the maximum probability value P of the pixel belonging to a cluster obtained by the last M steps_maxDetermining a region item segmentation parameter of the pixel, wherein the region item segmentation parameter is the probability that the pixel is judged to be a foreground point or a background point;

and M: determining the probability of each pixel belonging to each cluster according to the GMM model of each cluster, and for any pixel, determining the maximum probability value P of the pixel_maxThe corresponding cluster determines the classification of the pixel,

determining a segmentation parameter of each pixel on the mask map according to the depth map and the mask map, further comprising: determining boundary item segmentation parameters of the pixels according to the depth value difference between the pixels and adjacent pixels;

2. The method of claim 1, wherein:

constructing an undirected graph, and mapping the segmentation parameters of each pixel in the mask graph into the undirected graph, wherein the method comprises the following steps:

3. The method of claim 2, wherein:

the processing the undirected graph according to the minimum cut-maximum flow algorithm to obtain a finely divided mask graph comprises the following steps:

wherein, step C and step D include the following treatment respectively:

c, step C: for division and suspension point Q in undirected graph₀The foreground points of the same kind form a foreground point set Q by pixels divided into the foreground points;

4. The method of claim 1, wherein:

constructing a geometric figure containing the smearing track, expanding the geometric figure to form a marking area, and determining an area containing the smearing track as an interesting area, wherein the method comprises the following steps:

5. The method of claim 4, wherein:

the constructing of the minimum circumscribed rectangle of the smearing track, the expanding of the minimum circumscribed rectangle according to the length L of the smearing track to form a marking area, includes:

6. The method of claim 1, wherein:

before generating the input mask map of the image segmentation algorithm, the method further comprises the following steps:

7. An apparatus for enabling interactive image segmentation, comprising:

an output module, configured to segment an image corresponding to a foreground point in the finely segmented mask map from an original image obtained by capturing the target object,

8. A terminal comprising the apparatus for implementing interactive image segmentation as claimed in claim 7.