WO2017067390A1

WO2017067390A1 - Method and terminal for obtaining depth information of low-texture regions in image

Info

Publication number: WO2017067390A1
Application number: PCT/CN2016/101602
Authority: WO
Inventors: 戴向东
Original assignee: 努比亚技术有限公司
Priority date: 2015-10-20
Filing date: 2016-10-09
Publication date: 2017-04-27
Also published as: CN105354838B; CN105354838A

Abstract

A terminal and a method for obtaining depth information of the low-texture region in an image, the terminal comprising: an image segmentation module (301) configured to obtain the color and brightness information of the image, and to segment the image into a plurality of regions according to the color and brightness information; a low-texture region obtaining module (302) configured to calculate and obtain the gradient information corresponding to the image, and to select, according to the gradient information, a low-texture region from a plurality of regions segmented by the image segmentation module, the low-texture region being a region in which the statistical average gradient is within a preset range; an edge depth obtaining module (303) configured to extract boundary pixel points of the low-texture region selected by the low-texture region obtaining module (302), and to obtain the depth values of the boundary pixel points; and a region depth obtaining module (304) configured to calculate the depth value of each of the pixel points in the low-texture region according to the boundary pixel point depth value obtained by the edge depth obtaining module (303).

Description

Method and terminal for acquiring depth information of weak texture region in image

Technical field

The present invention relates to the field of image processing, and in particular, to a method and a terminal for acquiring depth information of a weak texture region in an image.

Background technique

Stereo matching is a core algorithm in stereo vision. In the stereo matching algorithm, the processing of weak texture regions has always been a difficult point. In many application scenarios, the pixels of weak texture regions are similar in color and brightness. The matching of the pixel points brings singularity, and the stereo matching algorithm is easy to mismatch, so that the depth information of the obtained weak texture region is not accurate.

Summary of the invention

In view of this, the embodiment of the present invention is to provide a method and a terminal for acquiring a depth information of a weak texture region in an image, which can accurately obtain depth information of a weak texture region.

In order to achieve the above object, the technical solution of the present invention is achieved as follows:

A terminal comprising:

An image segmentation module configured to acquire color and brightness information of an image, and divide the image into a plurality of regions according to the color and brightness information;

a weak texture region acquiring module configured to calculate gradient information corresponding to the image, and select a weak texture region from the regions segmented by the image segmentation module according to the gradient information, where the weak texture region is a gradient statistical average An area within a preset range;

An edge depth obtaining module, configured to extract boundary pixel points of the weak texture region selected by the weak texture region acquiring module, and obtain a depth value of the boundary pixel point;

a region depth obtaining module configured to acquire a boundary image obtained by the module according to the edge depth The depth value of the prime point, and the depth value of each pixel in the weak texture region is calculated.

In the above solution, the weak texture region obtaining module is further configured to: obtain gradient information corresponding to the image according to a gradient algorithm, where the gradient information is a gradient corresponding to each pixel in the image; and calculate the image. The gradient statistical average corresponding to the pixel points in the segmentation region is selected, and the region in which the gradient statistical average value is within the preset range is a weak texture region.

In the above solution, the region depth obtaining module is further configured to: screen out a sudden change point in the boundary pixel point according to a depth value of the boundary pixel point, to obtain a reliable point in the boundary pixel point; The depth value of the point is plane-fitted, and the depth value of each pixel in the weak texture area is calculated.

In the above solution, the image segmentation module is further configured to divide the image into a plurality of regions by using a region-based segmentation method, a threshold-based segmentation method, an edge-based segmentation method, or a cluster analysis method.

In the above solution, the image segmentation module is further configured to select a plurality of seed pixels, and divide a new pixel point around the seed pixel that meets a preset condition into an area where the seed pixel is located, and the new pixel The point is regarded as a new seed pixel, and the new pixel point satisfying the preset condition around the new seed pixel is further divided into an area where the seed pixel is located until the new seed pixel does not exist. Determining a plurality of regions divided according to the plurality of seed pixels; wherein the preset condition is that the difference between the color and the luminance information is at a first threshold compared to the seed pixel Inside.

In the above solution, the edge depth acquiring module is further configured to perform area marking on the weak texture area to obtain the marked weak texture area, and then binarize the marked weak texture area and other areas of the image. Obtaining a weakly textured region after binarization, performing the cavity filling of the binarized weakly textured region to obtain a weakly textured region after the cavity is filled, and performing contour extraction on the weakly textured region after the cavity is filled to obtain the a contour line of the weak texture region, and acquiring a boundary pixel point of the weak texture region according to the contour line of the weak texture region; Calculating the depth value Z of the boundary pixel:

Where f is the focal length of two digital cameras in the stereoscopic imaging device, T is the spacing between the two digital cameras, and d is the parallax corresponding to the boundary pixels in the disparity map of the two images taken by the two digital cameras value.

In the above solution, the area mark refers to: making the continuous area the same mark;

The edge depth obtaining module is further configured to perform a region marking on the weak texture region to obtain a marked weak texture region by using a four-neighbor labeling algorithm or an eight-neighbor labeling algorithm.

In the above solution, the edge depth acquiring module is further configured to adopt a region hole filling algorithm, and the binarized weak texture region is filled with the region hole to obtain the weak texture region after the cavity filling.

In the foregoing solution, the edge depth acquiring module is further configured to apply a stereo matching algorithm to obtain a depth value of the boundary pixel.

In the above solution, the area depth obtaining module is further configured to use the depth value of the boundary pixel as the sample set P, and randomly extract the depth values of the n boundary pixels from the sample set P as the subset S, and pass the The plane fitting obtains the initialization model M; the depth value of the boundary pixel points in the residual set SC=P\S and the error of the initialization model M is less than the second threshold is divided into inner point sets; when the inner point concentrates the number of depth values When N is reached, the new model M* is recalculated according to the inner point set by least squares method; the new subset S* is re-randomly extracted, and the above process is repeated; after repeating a certain number of times, the obtained maximum inner point set is selected. The depth value of the maximum inner point set is a depth value of a reliable point in the boundary pixel point, and the remaining value in the sample set P is a depth value of a sudden change point in the boundary pixel point, the n and N Is the default value.

A method for acquiring depth information of a weak texture region in an image, the method comprising:

Obtaining color and brightness information of the image, and dividing the image into a plurality of regions according to the color and brightness information;

Obtaining gradient information corresponding to the image, and selecting a weak texture region from the plurality of regions according to the gradient information, where the weak texture region is an area in which a gradient statistical average value is within a preset range;

Extracting boundary pixel points of the weak texture region, and acquiring a depth value of the boundary pixel point;

Determining a depth value of each pixel in the weak texture region according to the depth value of the boundary pixel.

In the above solution, the calculating obtains the gradient information corresponding to the image, and selects the weak texture region from the plurality of regions according to the gradient information, including:

Obtaining gradient information corresponding to the image according to a gradient algorithm, where the gradient information is a gradient corresponding to each pixel point in the image;

Calculating a gradient statistical average corresponding to the pixel points in the plurality of image segmentation regions, and selecting a region in which the gradient statistical average value is within the preset range is a weak texture region.

In the above solution, the calculating, according to the depth value of the boundary pixel, the depth value of each pixel in the weak texture region, including:

And filtering a sudden change point in the boundary pixel point according to the depth value of the boundary pixel point to obtain a reliable point in the boundary pixel point; performing plane fitting on the depth value of the reliable point to calculate the weak point The depth value of each pixel in the texture area.

In the above solution, the dividing the image into a plurality of regions according to the color and brightness information includes:

The image is segmented into a plurality of regions using a region-based segmentation method, a threshold-based segmentation method, or an edge-based segmentation method or a cluster analysis method.

In the above solution, the region-based segmentation method is used to divide the image into several regions, including:

Selecting a plurality of seed pixels, dividing a new pixel point satisfying a preset condition around the seed pixel point into an area where the seed pixel point is located, and treating the new pixel point as a new seed pixel Pointing, continuing to divide a new pixel point satisfying the preset condition around the new seed pixel to an area where the seed pixel point is located, until there is no pixel point satisfying the preset condition around the new seed pixel point, Obtaining a plurality of regions divided according to the plurality of seed pixels; wherein the preset condition is that a difference between the color and the luminance information is within a first threshold compared to the seed pixel.

In the above solution, the extracting the boundary pixel of the weak texture region and acquiring the depth value of the boundary pixel includes:

Performing area marking on the weak texture area to obtain the marked weak texture area, and then binarizing the marked weak texture area and other areas of the image to obtain a binarized weak texture area, and the second The weakened texture region is filled with the region to obtain the weakly textured region after the cavity is filled, and the weakly textured region after the cavity is filled for contour extraction to obtain the contour of the weakly textured region, according to the weak texture region The contour line acquires a boundary pixel point of the weak texture region; and calculates a depth value Z of the boundary pixel point according to the following formula:

And performing the area marking on the weak texture area to obtain the marked weak texture area, including:

The weak texture region is region-marked to obtain the marked weak texture region by using a four-neighbor labeling algorithm or an eight-neighbor labeling algorithm.

In the above solution, the null-textured region of the binarized weakly textured region is filled with a cavity to obtain a weakly textured region after the cavity is filled, including:

The area void filling algorithm is used to fill the weakly textured area of the binarized area to obtain the weakly textured area after the cavity is filled.

In the above solution, the obtaining the depth value of the boundary pixel includes:

A stereo matching algorithm is applied to obtain a depth value of the boundary pixel.

In the above solution, the filtering out the abrupt point in the boundary pixel according to the depth value of the boundary pixel to obtain a reliable point in the boundary pixel, including:

Taking the depth value of the boundary pixel as the sample set P, randomly extracting the depth values of the n boundary pixel points from the sample set P as the subset S, and obtaining the initialization model M by plane fitting; the residual set SC=P The depth value of the boundary pixel point in the \S with the initialization model M whose error is less than the second threshold is divided into inner point sets; when the number of inner point concentration depth values reaches N, the least square method is used according to the inner point set. Calculating a new model M*; re-randomly extracting a new subset S*, repeating the above process; after repeating a certain number of times, selecting the obtained maximum inner point set, the depth value of the largest inner point set is the boundary pixel The depth value of the reliable point in the point, the remaining value in the sample set P is the depth value of the abrupt point in the boundary pixel point, and the n and N are preset values.

An embodiment of the present invention provides a method and a terminal for acquiring depth information of a weak texture region in an image, where the terminal divides an image into a plurality of image segmentation regions; and obtains a weak texture region according to the gradient statistical detection; and extracts the weak texture region. a boundary pixel, applying a stereo matching algorithm to obtain a depth value of the boundary pixel; and filtering a sudden change point in the boundary pixel according to the depth value of the boundary pixel to obtain a reliable point in the boundary pixel And performing a plane fitting on the depth value of the reliable point to calculate a depth value of each pixel in the weak texture region. Compared with the depth value of each pixel in the weak texture region, the method of the present embodiment performs plane fitting according to the depth value of the selected reliable point, and estimates the weak texture region. The depth value of each pixel point can reduce the error probability of the depth value estimation of the weak texture region, and accurately obtain the depth information of the weak texture region.

DRAWINGS

1 is a schematic structural diagram of hardware of a mobile terminal that implements various embodiments of the present invention;

2 is a schematic diagram of a wireless communication system of the mobile terminal shown in FIG. 1;

3 is a structural block diagram of a terminal according to Embodiment 1 of the present invention;

4 is an image for processing according to an embodiment of the present invention;

FIG. 5 is a diagram of identifying, by using a plurality of color blocks, a plurality of image segmentation regions after the image shown in FIG. 4 is segmented according to an embodiment of the present invention;

FIG. 6 is an image showing a weak texture area according to an embodiment of the present invention;

FIG. 7 is an image obtained by binarizing a weak texture region according to an embodiment of the present invention;

FIG. 8 is an image of a binarized image shown in FIG. 7 after being filled in a cavity according to an embodiment of the present invention; FIG.

FIG. 9 is an image obtained by performing contour extraction on a hole-filled image shown in FIG. 8 according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of plane fitting of a depth value of a reliable point according to an embodiment of the present invention; FIG.

FIG. 11 is a depth image obtained by applying an existing method according to an embodiment of the present invention;

FIG. 12 is a depth image obtained by applying the method according to an embodiment of the present invention;

FIG. 13 is a schematic flowchart diagram of a method for acquiring depth information of a weak texture region in an image according to Embodiment 2 of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in the following with reference to the accompanying drawings.

A mobile terminal embodying various embodiments of the present invention will now be described with reference to FIG. In the following description, the use of suffixes such as "module", "component" or "unit" for indicating an element is merely an explanation for facilitating the present invention, and does not have a specific meaning per se. Therefore, "module" and "component" can be used in combination.

The mobile terminal can be implemented in various forms. For example, the terminal described in the present invention may include, for example, a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, and a personal digital assistant. Mobile terminals of PDAs, tablet computers (PADs), portable multimedia players (PMPs), navigation devices, and the like, and fixed terminals such as digital TVs, desktop computers, and the like. In the following, it is assumed that the terminal is a mobile terminal. However, those skilled in the art will appreciate that configurations in accordance with embodiments of the present invention can be applied to fixed type terminals in addition to components that are specifically for mobile purposes.

FIG. 1 is a schematic diagram showing the hardware structure of a mobile terminal embodying various embodiments of the present invention.

The mobile terminal 100 may include a wireless communication unit 110, an audio/video (A/V) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, an interface unit 170, a controller 180, and a power supply unit 190. and many more. Figure 1 shows a mobile terminal having various components, but it should be understood that not all illustrated components are required to be implemented, and more or fewer components may be implemented instead, and the components of the mobile terminal will be described in detail below. .

Wireless communication unit 110 typically includes one or more components that permit radio communication between mobile terminal 100 and a wireless communication system or network. For example, the wireless communication unit may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless internet module 113, a short-range communication module 114, and a location information module 115.

The broadcast receiving module 111 receives a broadcast signal and/or broadcast associated information from an external broadcast management server via a broadcast channel. The broadcast channel can include a satellite channel and/or a terrestrial channel. The broadcast management server may be a server that generates and transmits a broadcast signal and/or broadcast associated information or a server that receives a previously generated broadcast signal and/or broadcast associated information and transmits it to the terminal. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and the like. Moreover, the broadcast signal may further include a broadcast signal combined with a TV or radio broadcast signal. The broadcast associated information may also be provided via a mobile communication network, and in this case, the broadcast associated information may be received by the mobile communication module 112. The broadcast signal may exist in various forms, for example, it may exist in the form of Digital Multimedia Broadcasting (DMB) Electronic Program Guide (EPG), Digital Video Broadcasting Handheld (DVB-H) Electronic Service Guide (ESG), and the like. . Broadcast receiving module The signal broadcast can be received by using various types of broadcast systems. In particular, the broadcast receiving module 111 can use forward link media (Media) such as Multimedia Broadcast-Ground (DMB-T), Digital Multimedia Broadcast-Satellite (DMB-S), Digital Video Broadcast-Handheld (DVB-H) Digital broadcasting systems such as FLO@) data broadcasting systems, terrestrial digital broadcasting integrated services (ISDB-T), etc. receive digital broadcasting. The broadcast receiving module 111 can be constructed as various broadcast systems suitable for providing broadcast signals as well as the above-described digital broadcast system. The broadcast signal and/or broadcast associated information received via the broadcast receiving module 111 may be stored in the memory 160 (or other type of storage medium).

The mobile communication module 112 transmits the radio signals to and/or receives radio signals from at least one of a base station (e.g., an access point, a Node B, etc.), an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data transmitted and/or received in accordance with text and/or multimedia messages.

The wireless internet module 113 supports wireless internet access of the mobile terminal. The module can be internally or externally coupled to the terminal. The wireless Internet access technologies involved in the module may include Wireless Local Area Network (WLAN) (Wi-Fi), Wireless Broadband (Wibro), Worldwide Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), and the like. .

The short range communication module 114 is a module for supporting short range communication. Some examples of short-range communication technologies include BluetoothTM, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigbeeTM, and the like.

The location information module 115 is a module for checking or acquiring location information of the mobile terminal. A typical example of a location information module is the Global Positioning System (GPS). According to the current technology, the GPS module 115 calculates distance information and accurate time information from three or more satellites and applies triangulation to the calculated information to accurately calculate three-dimensional current position information based on longitude, latitude, and altitude. Currently, the method for calculating position and time information uses three satellites and corrects the calculated position and time information errors by using another satellite. Further, the GPS module 115 is capable of calculating speed information by continuously calculating current position information in real time.

The A/V input unit 120 is for receiving an audio or video signal. The A/V input unit 120 may include a camera 121 and a microphone 122 that processes image data of still pictures or video obtained by the image capturing device in a video capturing mode or an image capturing mode. The processed image frame can be displayed on the display unit 151. The image frames processed by the camera 121 may be stored in the memory 160 (or other storage medium) or transmitted via the wireless communication unit 110, and two or more cameras 121 may be provided according to the configuration of the mobile terminal. The microphone 122 can receive sound (audio data) via a microphone in an operation mode of a telephone call mode, a recording mode, a voice recognition mode, and the like, and can process such sound as audio data. The processed audio (voice) data can be converted to a format output that can be transmitted to the mobile communication base station via the mobile communication module 112 in the case of a telephone call mode. The microphone 122 can implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated during the process of receiving and transmitting audio signals.

The user input unit 130 may generate key input data according to a command input by the user to control various operations of the mobile terminal. The user input unit 130 allows the user to input various types of information, and may include a keyboard, a pot, a touch pad (eg, a touch sensitive component that detects changes in resistance, pressure, capacitance, etc. due to contact), a scroll wheel , rocker, etc. In particular, when the touch panel is superimposed on the display unit 151 in the form of a layer, a touch screen can be formed.

The sensing unit 140 detects the current state of the mobile terminal 100 (eg, the open or closed state of the mobile terminal 100), the location of the mobile terminal 100, the presence or absence of contact (ie, touch input) by the user with the mobile terminal 100, and the mobile terminal. The orientation of 100, the acceleration or deceleration movement and direction of the mobile terminal 100, and the like, and generates a command or signal for controlling the operation of the mobile terminal 100. For example, when the mobile terminal 100 is implemented as a slide type mobile phone, the sensing unit 140 can sense whether the slide type phone is turned on or off. In addition, the sensing unit 140 can detect whether the power supply unit 190 provides power or whether the interface unit 170 is coupled to an external device. Sensing unit 140 may include proximity sensor 141 which will be described below in connection with a touch screen.

The interface unit 170 serves as an interface through which at least one external device can connect with the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, and an audio input/output. (I/O) port, video I/O port, headphone port, and more. The identification module may be stored to verify various information used by the user using the mobile terminal 100 and may include a User Identification Module (UIM), a Customer Identification Module (SIM), a Universal Customer Identity Module (USIM), and the like. In addition, the device having the identification module (hereinafter referred to as "identification device") may take the form of a smart card, and thus the identification device may be connected to the mobile terminal 100 via a port or other connection device. The interface unit 170 can be configured to receive input from an external device (eg, data information, power, etc.) and transmit the received input to one or more components within the mobile terminal 100 or can be used at the mobile terminal and external device Transfer data between.

In addition, when the mobile terminal 100 is connected to the external base, the interface unit 170 may function as a path through which power is supplied from the base to the mobile terminal 100 or may be used as a transmission of various command signals allowing input from the base to the mobile terminal 100 The path to the terminal. Various command signals or power input from the base can be used as signals for identifying whether the mobile terminal is accurately mounted on the base. Output unit 150 is configured to provide an output signal (eg, an audio signal, a video signal, an alarm signal, a vibration signal, etc.) in a visual, audio, and/or tactile manner. The output unit 150 may include a display unit 151, an audio output module 152, an alarm unit 153, and the like.

The display unit 151 can display information processed in the mobile terminal 100. For example, when the mobile terminal 100 is in a phone call mode, the display unit 151 can display a user interface (UI) or a graphical user interface (GUI) related to a call or other communication (eg, text messaging, multimedia file download, etc.). When the mobile terminal 100 is in a video call mode or an image capturing mode, the display unit 151 may display a captured image and/or a received image, a UI or GUI showing a video or image and related functions, and the like.

Meanwhile, when the display unit 151 and the touch panel are superposed on each other in the form of a layer to form a touch screen, The display unit 151 can be used as an input device and an output device. The display unit 151 may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), an organic light emitting diode (OLED) display, a flexible display, a three-dimensional (3D) display, and the like. Some of these displays may be configured to be transparent to allow a user to view from the outside, which may be referred to as a transparent display, and a typical transparent display may be, for example, a TOLED (Transparent Organic Light Emitting Diode) display or the like. According to a particular desired embodiment, the mobile terminal 100 may include two or more display units (or other display devices), for example, the mobile terminal may include an external display unit (not shown) and an internal display unit (not shown) . The touch screen can be used to detect touch input pressure as well as touch input position and touch input area.

The audio output module 152 can convert the audio data received by the wireless communication unit 110 or stored in the memory 160 when the mobile terminal is in a call signal receiving mode, a call mode, a recording mode, a voice recognition mode, a broadcast receiving mode, and the like. The audio signal is output as sound. Moreover, the audio output module 152 can provide audio output (eg, call signal reception sound, message reception sound, etc.) associated with a particular function performed by the mobile terminal 100. The audio output module 152 can include a speaker, a buzzer, and the like.

The alarm unit 153 can provide an output to notify the mobile terminal 100 of the occurrence of an event. Typical events may include call reception, message reception, key signal input, touch input, and the like. In addition to audio or video output, the alert unit 153 can provide an output in a different manner to notify of the occurrence of an event. For example, the alarm unit 153 can provide an output in the form of vibrations, and when a call, message, or some other incoming communication is received, the alarm unit 153 can provide a tactile output (ie, vibration) to notify the user of it. By providing such a tactile output, the user is able to recognize the occurrence of various events even when the user's mobile phone is in the user's pocket. The alarm unit 153 can also provide an output of the notification event occurrence via the display unit 151 or the audio output module 152.

The memory 160 can store a software program that is processed and controlled by the controller 180. And so on, or data that has been output or is about to be output (for example, a phone book, a message, a still image, a video, etc.) can be temporarily stored. Moreover, the memory 160 can store data regarding vibrations and audio signals of various manners that are output when a touch is applied to the touch screen.

The memory 160 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), a random access memory (RAM), a static random access memory ( SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like. Moreover, the mobile terminal 100 can cooperate with a network storage device that performs a storage function of the memory 160 through a network connection.

The controller 180 typically controls the overall operation of the mobile terminal. For example, the controller 180 performs the control and processing associated with voice calls, data communications, video calls, and the like. In addition, the controller 180 may include a multimedia module 181 for reproducing (or playing back) multimedia data, which may be constructed within the controller 180 or may be configured to be separate from the controller 180. The controller 180 may perform a pattern recognition process to recognize a handwriting input or a picture drawing input performed on the touch screen as a character or an image.

The power supply unit 190 receives external power or internal power under the control of the controller 180 and provides appropriate power required to operate the various components and components.

The various embodiments described herein can be implemented in a computer readable medium using, for example, computer software, hardware, or any combination thereof. For hardware implementations, the embodiments described herein may be through the use of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays ( An FPGA, a processor, a controller, a microcontroller, a microprocessor, at least one of the electronic units designed to perform the functions described herein, in some cases, such an embodiment may be at the controller 180 Implemented in the middle. For software implementations, implementations such as procedures or functions may be implemented with separate software modules that permit the execution of at least one function or operation. Software code can be used by The software application (or program) written in a suitable programming language is implemented, and the software code can be stored in the memory 160 and executed by the controller 180.

So far, the mobile terminal has been described in terms of its function. Hereinafter, for the sake of brevity, a slide type mobile terminal among various types of mobile terminals such as a folding type, a bar type, a swing type, a slide type mobile terminal, and the like will be described as an example. Therefore, the present invention can be applied to any type of mobile terminal, and is not limited to a slide type mobile terminal.

The mobile terminal 100 as shown in FIG. 1 may be configured to operate using a communication system such as a wired and wireless communication system and a satellite-based communication system that transmits data via frames or packets.

A communication system in which a mobile terminal according to the present invention can be operated will now be described with reference to FIG.

Such communication systems can use different air interfaces and/or physical layers. For example, air interfaces used by communication systems include, for example, Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), and Universal Mobile Telecommunications System (UMTS) (in particular, Long Term Evolution (LTE)). ), Global System for Mobile Communications (GSM), etc. As a non-limiting example, the following description relates to CDMA communication systems, but such teachings are equally applicable to other types of systems.

Referring to FIG. 2, a CDMA wireless communication system may include a plurality of mobile terminals 100, a plurality of base stations (BS) 270, a base station controller (BSC) 275, and a mobile switching center (MSC) 280. The MSC 280 is configured to interface with a public switched telephone network (PSTN) 290. The MSC 280 is also configured to interface with a BSC 275 that can be coupled to the base station 270 via a backhaul line. The backhaul line can be constructed in accordance with any of a number of well known interfaces including, for example, E1/T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL, or xDSL. It will be appreciated that the system as shown in FIG. 2 can include multiple BSCs 275.

Each BS 270 can serve one or more partitions (or regions), each of which is covered by a multi-directional antenna or an antenna directed to a particular direction radially away from the BS 270. Alternatively, each partition may be covered by two or more antennas for diversity reception. Each BS 270 can be configured to support multiple frequency allocations, and each frequency allocation has a particular frequency spectrum (eg, 1.25 MHz, 5 MHz, etc.) Wait).

The intersection of partitioning and frequency allocation can be referred to as a CDMA channel. BS 270 may also be referred to as a Base Transceiver Subsystem (BTS) or other equivalent terminology. In such a case, the term "base station" can be used to generally refer to a single BSC 275 and at least one BS 270. A base station can also be referred to as a "cell station." Alternatively, each partition of a particular BS 270 may be referred to as a plurality of cellular stations.

As shown in FIG. 2, a broadcast transmitter (BT) 295 transmits a broadcast signal to the mobile terminal 100 operating within the system. A broadcast receiving module 111 as shown in FIG. 1 is provided at the mobile terminal 100 to receive a broadcast signal transmitted by the BT 295. In Figure 2, several Global Positioning System (GPS) satellites 300 are shown. The satellite 300 helps locate at least one of the plurality of mobile terminals 100.

In Figure 2, a plurality of satellites 300 are depicted, but it is understood that useful positioning information can be obtained using any number of satellites. The GPS module 115 as shown in Figure 1 is typically configured to cooperate with the satellite 300 to obtain desired positioning information. Instead of GPS tracking technology or in addition to GPS tracking technology, other techniques that can track the location of the mobile terminal can be used. Additionally, at least one GPS satellite 300 can selectively or additionally process satellite DMB transmissions.

As a typical operation of a wireless communication system, BS 270 receives reverse link signals from various mobile terminals 100. Mobile terminal 100 typically participates in calls, messaging, and other types of communications. Each reverse link signal received by a particular base station 270 is processed within a particular BS 270. The obtained data is forwarded to the relevant BSC 275. The BSC provides call resource allocation and coordinated mobility management functions including a soft handoff procedure between the BSs 270. The BSC 275 also routes the received data to the MSC 280, which provides additional routing services for interfacing with the PSTN 290. Similarly, PSTN 290 interfaces with MSC 280, which forms an interface with BSC 275, and BSC 275 controls BS 270 accordingly to transmit forward link signals to mobile terminal 100.

Based on the above-described mobile terminal hardware structure and communication system, various embodiments of the method of the present invention are proposed.

Example 1

The embodiment of the present invention provides a terminal. As shown in FIG. 3, the terminal includes an image segmentation module 301, a weak texture region acquisition module 302, an edge depth acquisition module 303, and a region depth acquisition module 304.

The image segmentation module 301 is configured to acquire color and brightness information of the image, and divide the image into a plurality of regions according to the color and brightness information.

For the image shown in FIG. 4, the image segmentation module 301 can segment the image according to the color and brightness information of the image.

The image segmentation module 301 may specifically apply a region-based segmentation method such as a region growing method to segment an image. The basic idea of region growing is to group pixels with similar properties to form regions. Specifically, a seed pixel is searched for each of the regions to be segmented as a starting point for growth, and then pixels in the periphery of the seed pixel having the same or similar properties as the seed pixel (in this embodiment, pixels having similar color and luminance information) ) merged into the area where the seed pixel is located. These new pixels are treated as new seed pixels to continue the above process until no more pixels satisfying the condition can be included. Such an area will grow.

That is, the image segmentation module is further configured to select a plurality of seed pixels, divide a new pixel point satisfying the preset condition around the seed pixel point into an area where the seed pixel point is located, and treat the new pixel point as a new seed pixel, continuing to divide a new pixel point satisfying the preset condition around the new seed pixel to an area where the seed pixel point is located, until a preset condition does not exist around the new seed pixel point a pixel, obtaining a plurality of regions divided according to the plurality of seed pixels; wherein the preset condition is that a difference between the color and the luminance information is within a first threshold compared to the seed pixel.

The image segmentation module may further divide the image according to color and brightness information of the image by using a mean shift algorithm to acquire a plurality of regions.

Mean Shift algorithm is an effective statistical iterative algorithm. Image segmentation based on Mean Shift algorithm is also a region-based segmentation method. The analysis features are very similar and have strong adaptability and robustness. It is not sensitive to the smooth area of the image and the image texture area, so it can get good segmentation results. This algorithm has been widely used in the field of computer vision and has achieved great success. This embodiment may apply the Mean Shift algorithm to segment the image into a plurality of image segmentation regions according to color and luminance information.

The approximate steps for image segmentation using the Mean Shift algorithm are as follows:

1. For each pixel point i, initialize j=1 and make y _i,1 =x _i ;

2. Calculate y _i,j+1 using the mean shift algorithm, ie

Where w(x _i ) is the weight coefficient, g(x)=-k'(x) is called the shadow function of k, k is the kernel function profile function, and the process of mean shift is continuously performed for each feature vector x _i , converge to different mode points through multiple iterations, the value after convergence is y _ic , assignment

3. Repeat steps 1 and 2 to form a cluster center set C _d ={c _d,k ,k=1,2,.....n}; after the pre-classification process, the initial feature vector is clustered. The center is divided into n categories.

4. Then, C _d is detected from the airspace. If any c _i , c _j ∈ C _d , i ≠ j satisfies in the same bounding sphere in the feature space, the features are considered to be similar, and c _i and c _{j are} classified into one. The class, that is, after the above processing, the pixels that are finally clustered into the same class are divided into a region, so that the image is divided into several regions.

Of course, in the case of image-based color and luminance information, the image segmentation module 301 may further divide the image by using a threshold-based segmentation method, an edge-based segmentation method, a clustering segmentation method, or the like. The specific image segmentation method adopted is not limited herein.

The image segmentation module 301 performs image segmentation based on color and brightness information of the image. The pixels inside the divided area are similar in color and brightness. For example, the image shown in FIG. 4 is segmented, and different image segmentation regions are represented by different color blocks, and the effect is as shown in FIG. 5.

The weak texture region obtaining module 302 is configured to calculate the gradient information corresponding to the image, and select a weak texture region from the regions segmented by the image segmentation module according to the gradient information, where the weak texture region is a gradient statistical average An area whose value is within the preset range.

The image can be regarded as a two-dimensional discrete function I(i,j), (i,j) is the coordinates of the pixel points in the image, and I(i,j) is the pixel value of the pixel point (i,j) (eg RGB) Value), the gradient information of the image is actually the derivation of this two-dimensional discrete function:

The gradient information of the image may be: G(x, y) = dx(i, j) + dy(i, j);

Where dx(i,j)=I(i+1,j)-I(i,j);

Dy(i,j)=I(i,j+1)-I(i,j);

You can also use the median difference:

Dx(i,j)=[I(i+1,j)-I(i-1,j)]/2;

Dy(i,j)=[I(i,j+1)-I(i,j-1)]/2;

The above is just an example of the simplest gradient definition, but there are more and more complex gradient formulas. For example: Sobel, Roberts, kirsch, laplace, piewitt, robinson operators, etc.

The gradient size of the image can reflect the brightness of the pixels of the image and the frequency change of the color. For the weak texture region, the brightness of the internal pixel points is similar, the change is small, and the corresponding gradient value is relatively small. According to the principle, For the regions into which the image segmentation module 301 is divided, the region in which the gradient statistical average is small is the weak texture region.

In this embodiment, the weak texture region obtaining module 302 may calculate the gradient information corresponding to the image according to an existing gradient algorithm, that is, obtain a gradient corresponding to each pixel point in the image, and then obtain the weak texture region. The module 302 may calculate a gradient statistical average value corresponding to the pixel points in the plurality of regions divided by the image segmentation module 301, and select a gradient statistical average value. The area within the preset range is a weakly textured area. The preset range is a range in which the gradient statistical average value is small, such as 0-10, which may be specifically determined according to actual conditions.

By way of example, for the image shown in FIG. 4, the three weak texture regions as illustrated in FIG. 6 can be obtained by the above-described processing of the weak texture region acquisition module 302.

The edge depth obtaining module 303 is configured to extract boundary pixel points of the weak texture region selected by the weak texture region obtaining module 302, and obtain a depth value of the boundary pixel point.

The edge depth obtaining module 303 is further configured to perform area marking on the weak texture area to obtain a marked weak texture area, and then binarize the marked weak texture area and other areas of the image to obtain a binary value. a weak texture region after the binarized weak texture region is filled with the region to obtain a weakly textured region after the cavity is filled, and the weakly textured region after the cavity is filled for contour extraction to obtain the weak texture region a contour line, the boundary pixel of the weak texture region is obtained according to the contour of the weak texture region; and the depth value Z of the boundary pixel is obtained according to the following formula:

The area mark is to mark the continuous area with the same mark. The common area mark method has four neighborhood mark algorithm and eight neighborhood mark algorithm.

Binarization is to set the gray value of the pixel on the image to 0 or 255, that is, to present the entire image with a distinct black and white visual effect.

For the weak texture area located at the upper right of the image among the three weak texture areas indicated in FIG. 6, the area mark is performed, that is, the weak texture area on the upper right of the image is marked as the area No. 1, and the other areas are marked as the area No. 2; The marked weak texture region, that is, the No. 1 region is binarized with the other region of the image, that is, the No. 2 region, as shown in FIG. 7, the weak texture located at the upper right of the image. The area is displayed in white and the other areas in the image are displayed in black. Usually, due to the influence of noise, some of the pixels in the weak texture region are mistakenly considered to be not in the weak texture region, resulting in voids in the detected weak texture region, such as the black dots in the white region in FIG.

In order to obtain the boundary pixel points of the weak texture region, the edge depth acquisition module 303 uses the region hole filling algorithm to perform the region hole filling on the binarized weak texture region to obtain the void-filled weak texture region. The image after the area filling is completed in the weak texture area is shown in Fig. 8.

At this time, the contour extraction can be performed on FIG. 8. Since the black and white contrast is clearly defined, the edge depth acquisition module 303 can easily extract the outline of the weak texture region, and the outline thereof is as shown by the line in FIG. The boundary pixel points of the weak texture region can be acquired according to the contour line of the weak texture region.

After the edge depth obtaining module 303 extracts the boundary pixel points of the weak texture region, a stereo matching algorithm may be applied to obtain the depth value of the boundary pixel point.

The input of the stereo matching algorithm is an image acquired by a plurality of digital cameras of different viewing angles, and the output is a correspondence of points on the images. Assuming a geometric model of binocular stereo vision in a standard configuration, c and c' are the optical centers of the two cameras, f is the focal length, and T is the line connecting the two optical centers, that is, the spacing between the two digital cameras, also called the baseline. A line that passes through the optical center and is perpendicular to the imaging plane is called the optical axis. The so-called standard configuration means that the optical axes of the two cameras are perpendicular to the baseline and parallel to each other. Let the focal lengths of the two cameras be equal to f, and the horizontal coordinate of the coordinate system of the camera is parallel to the baseline direction, then the point P in the space has the same vertical coordinate on the images formed by the two cameras. This feature is also called stereoscopic vision. The Epipolar Line (the so-called outer pole line refers to the intersection of the outer pole plane and the image plane, where the outer pole plane is a plane containing two focal points and spatial points). For cameras with a typical configuration, images can be obtained in standard configuration by camera calibration and registration. The image after P point projection to the two cameras is x and x', respectively, and x and x' are a pair of corresponding points. If x and x' are used to represent their horizontal coordinates, the correspondence between the two points can be described by the parallax defined as follows:

Parallax d=x-x'

By deriving a simple geometric relationship, we can get the following equation:

Where Z represents the depth of the corresponding point.

It can be seen that when the baseline and the focal length are fixed, that is, the parameters of the camera and the relative position and posture between the cameras are fixed, the parallax d is inversely proportional to the depth Z of the point of the space. Therefore, it is only necessary to know the parallax of the pixel to obtain the depth of the pixel.

The edge depth acquisition module 303 may obtain the disparity d of the boundary pixel points of the weak texture region using a stereo matching algorithm, and calculate the Z of the boundary pixel point of the weak texture region using the following formula:

When the stereo matching algorithm is used to calculate the depth value of the pixel points in the weak texture region, since the pixels in the weak texture region are similar in color and brightness, this brings singularity to the matching of the pixel points, and the stereo matching algorithm is easy to apply. Mismatching, the depth information of the weak texture regions thus obtained is not accurate. However, for the boundary pixel points of the weak texture region, the pixel points inside the boundary pixel and the weak texture region are different in color and brightness, and the stereo matching algorithm is not easy to mismatch, so the boundary pixels of the obtained weak texture region are obtained. The depth value of the point is more accurate.

The area depth obtaining module 304 is configured to calculate a depth value of each pixel point in the weak texture area according to the depth value of the boundary pixel point acquired by the edge depth acquiring module 303.

The region depth obtaining module 304 may directly calculate the depth value of each pixel in the weak texture region according to the depth value of the boundary pixel point acquired by the edge depth acquiring module 303.

Due to the weakness of the stereo matching algorithm itself and the occlusion problem of the boundary point, abrupt pixel points with inaccurate partial depth values may occur. In this case, in order to obtain a more accurate depth value, the regional depth obtaining module 304 is further configured to The depth value of the boundary pixel is filtered to remove the abrupt point in the boundary pixel to obtain a reliable point in the boundary pixel; the depth value of the reliable point is plane-fitted, and the weak texture region is calculated The depth value of each pixel.

The region depth obtaining module 304 may specifically filter out a sudden change point in the boundary pixel by using a RANSAC algorithm.

The RANSAC (RANdom SAmple Consensus) algorithm is an algorithm for calculating valid mathematical sample parameters based on a set of sample data sets containing abnormal data. The RANSAC algorithm is often used in computer vision.

The basic assumption of the RANSAC algorithm is that the sample contains the correct data (inliers), which can be described by the model. It also contains outliers, that is, data that is far from the normal range and cannot adapt to the mathematical model, that is, the data set contains noise. These anomalous data may be due to erroneous measurements, incorrect assumptions, incorrect calculations, and the like. At the same time, RANSAC also assumes that given a correct set of data, there is a way to calculate the model parameters that match those data.

The basic idea of the RANSAC algorithm is described as follows:

1 Consider a model with a minimum sampling set of potential n (n is the minimum number of samples required to initialize the model parameters) and a sample set P, the number of samples of the set P#(P)>n, randomly extracted from P containing n a subset of P of the sample S initializes the model M;

2 The set of samples in SC=P\S with the error of model M being less than a certain set threshold t and S constitute S*. S* is considered to be an inner set of points, which constitute a Consistus Set of S;

3 If #(S*)≥N, think that the correct model parameters are obtained, and use the set S* (inners inliers) to recalculate the new model M* by least squares method; re-randomly extract new S, repeat above process.

4 After completing a certain number of sampling times, if the consistency set is not found, the algorithm fails. Otherwise, the largest uniform set obtained after sampling is selected to judge the inner and outer points, and the algorithm ends.

In this embodiment, the region depth obtaining module 304 is further configured to use the depth value of the boundary pixel as the sample set P, and randomly extract the depth values of the n boundary pixel points from the sample set P as the subset S. And obtaining an initialization model M by plane fitting; dividing a depth value of a boundary pixel point in the residual set SC=P\S that is smaller than a second threshold of the initialization model M into an inner point set; When the number of depth values in the inner point reaches N, the new model M* is recalculated according to the inner point set by least squares method; the new subset S* is re-randomly extracted, and the above process is repeated; After -10 times, selecting the obtained maximum inner point set, the depth value in the maximum inner point set is the depth value of the reliable point in the boundary pixel point, and the remaining values in the sample set P are The depth value of the abrupt point in the boundary pixel, the n and N being preset values. n is a preset value, which can be 60%-80% of the number of sample points in P, and the N value is also a preset value, which can be 90% of the number of sample points in P.

The region depth obtaining module 304 may perform plane fitting on the depth value of the reliable point to obtain a plane fitting equation, and the depth value of each pixel point in the weak texture region may be calculated by using the plane fitting equation. As shown in FIG. 10, a plane fitting diagram of the depth values of the boundary pixel points of a weak texture region, it can be seen that the plane of the boundary point fitting covers the weak texture region, and the depth values of other pixels in the region are It can be calculated by the plane fitting equation.

In practical applications, the functions implemented by each module unit in the terminal may be implemented by a central processing unit (CPU), a microprocessor (Micro Processor Unit, MPU), or a digital signal located in the terminal. Implemented by a processor (Digital Signal Processor, DSP) or a Field Programmable Gate Array (FPGA).

11 is a depth image obtained by applying the existing stereo matching algorithm, and FIG. 12 is a depth image of a weak texture region obtained by applying the method provided by the embodiment, and a depth image of another region obtained by applying the existing stereo matching algorithm; It is seen that the depth map of the weakly textured region processed by the mark in Fig. 6 becomes smooth, and the correct rate is improved.

The terminal of the present embodiment can accurately obtain the depth information of the weak texture region, so that when the background image is blurred by the depth image, in the weak texture region, the depth value estimation of the weak texture region occurs due to the singularity of the stereo matching algorithm. Errors, the effect of background blur will be affected, and the blur effect will be unnatural. By using the terminal of the embodiment to perform depth value estimation of the weak texture region, the error probability of the depth value estimation of the weak texture region can be reduced, and the background blur effect is better. In Lee When the target target region is estimated by the depth image, if the target region is a weak texture region, the target distance estimated by using the depth value of the singularity in the matching algorithm will be erroneous, and the depth value of the weak texture region is performed by using the terminal of the embodiment. It is estimated that the error probability of the depth value estimation of the weak texture region can be reduced, and the distance estimation of the target is more accurate; when the image segmentation is performed by using the depth image, if the target and the background both contain the texture region, the singularity in the matching algorithm is utilized. The area where the depth value is divided may be inaccurate. The depth value estimation of the weak texture area by using the terminal of the embodiment may reduce the error probability of the depth value estimation of the weak texture area, so that the image segmentation area is more accurate.

Example 2

An embodiment of the present invention provides a method for acquiring depth information of a weak texture region in an image. As shown in FIG. 13, the processing procedure of the method in this embodiment includes the following steps:

Step 1301: Acquire color and brightness information of an image, and divide the image into several regions according to the color and brightness information.

The terminal may apply a region-based segmentation method such as a region growing method to segment the image. The basic idea of region growing is to group pixels with similar properties to form regions. Specifically, a seed pixel is searched for each of the regions to be segmented as a starting point for growth, and then pixels in the periphery of the seed pixel having the same or similar properties as the seed pixel (in this embodiment, pixels having similar color and luminance information) ) merged into the area where the seed pixel is located. These new pixels are treated as new seed pixels to continue the above process until no more pixels satisfying the condition can be included. Such an area will grow.

That is, dividing the image into a plurality of regions according to the color and brightness information, specifically: selecting a plurality of seed pixel points, and dividing a new pixel point satisfying a preset condition around the seed pixel point to the seed pixel point And treating the new pixel point as a new seed pixel, and continuing to divide a new pixel point satisfying the preset condition around the new seed pixel into an area where the seed pixel is located until the new There is no preset condition around the seed pixel a pixel, obtaining a plurality of regions divided according to the plurality of seed pixels; wherein the preset condition is that a difference between the color and the luminance information is within a first threshold compared to the seed pixel.

The terminal can also use the meanshift algorithm to segment the image according to the color and brightness information of the image to obtain several regions.

Mean Shift algorithm is an effective statistical iterative algorithm. Image segmentation based on Mean Shift algorithm is also a region-based segmentation method. This segmentation method is very similar to the human eye's image analysis characteristics and has strong adaptability. Sex and robustness. It is not sensitive to the smooth area of the image and the image texture area, so it can get good segmentation results. This algorithm has been widely used in the field of computer vision and has achieved great success. This embodiment may apply the Mean Shift algorithm to segment the image into a plurality of image segmentation regions according to color and luminance information.

Of course, in the case of image-based color and luminance information, the terminal may also use a threshold-based segmentation method, an edge-based segmentation method, a cluster analysis method, and the like to divide the image, and the specific embodiment is adopted. The image segmentation method is not limited here.

In the method of the embodiment, the terminal performs image segmentation based on the color and brightness information of the image, and the internal pixels of the segmented image segmentation region are similar in color and brightness. For example, the image shown in FIG. 4 is segmented, and different image segmentation regions are represented by different color blocks, and the effect is as shown in FIG. 5.

Step 1302: Calculate the gradient information corresponding to the image, and select a weak texture region from the plurality of regions according to the gradient information.

The weakly textured region is an area in which the gradient statistical average is within a preset range.

The gradient information of the image may be: G(x, y) = dx(i, j) + dy(i, j);

Where dx(i,j)=I(i+1,j)-I(i,j);

Dy(i,j)=I(i,j+1)-I(i,j);

You can also use the median difference:

Dx(i,j)=[I(i+1,j)-I(i-1,j)]/2;

Dy(i,j)=[I(i,j+1)-I(i,j-1)]/2;

The gradient size of the image can reflect the brightness of the pixels of the image and the frequency change of the color. For the weak texture region, the brightness of the internal pixel points is similar, the change is small, and the corresponding gradient value is relatively small. According to the principle, For these regions into which the image is divided, the region in which the gradient statistical average is small is the weak texture region.

In this embodiment, the terminal may calculate the gradient information corresponding to the image according to a certain gradient algorithm, that is, obtain a gradient corresponding to each pixel in the image, and then calculate pixels in the image segmentation region. The gradient statistical average corresponding to the point is selected as the weak texture region in the region where the gradient statistical average is within the preset range. The preset range is a gradient statistical average value, such as 0-10, which can be defined according to the actual situation.

By way of example, for the image shown in Figure 4, there are three weakly textured regions as illustrated in Figure 6.

Step 1303: Extract boundary pixel points of the weak texture region, and obtain a depth value of the boundary pixel point.

The terminal may first perform area marking on the weak texture area to obtain the marked weak texture area, and then binarize the marked weak texture area and other areas of the image to obtain a binarized weak texture area, and The binarized weakly textured region is filled with the region to obtain a weakly textured region after the cavity is filled, and the weakly textured region after the cavity is filled for contour extraction to obtain the contour of the weakly textured region, according to the weak Obtaining a boundary pixel of the weak texture region; calculating a depth of the boundary pixel according to the following formula Value Z:

For the weak texture area located at the upper right of the image among the three weak texture areas indicated in FIG. 6, the area mark is performed, that is, the weak texture area on the upper right of the image is marked as the area No. 1, and the other areas are marked as the area No. 2; The marked weak texture region, that is, the No. 1 region, is binarized with the other region of the image, that is, the No. 2 region. As shown in FIG. 7, the weakly textured region located at the upper right of the image is displayed in white, and other regions in the image are displayed. Displayed in black. Usually, due to the influence of noise, some of the pixels in the weak texture region are mistakenly considered to be not in the weak texture region, resulting in voids in the detected weak texture region, such as the black dots in the white region in FIG.

In order to obtain the boundary pixel points of the weak texture region, the terminal uses the region hole filling algorithm to perform the hole filling of the binarized weak texture region to obtain the weak texture region after the cavity filling. The image after the area filling is completed in the weak texture area is shown in Fig. 8.

At this time, the contour extraction can be performed on FIG. 8. Since the black and white contrast is clearly defined, the terminal can easily extract the outline of the weak texture region, and the outline thereof is as shown by the line in FIG. The boundary pixel points of the weak texture region can be acquired according to the contour line of the weak texture region.

The obtaining the depth value of the boundary pixel includes: applying a stereo matching algorithm to obtain a depth value of the boundary pixel.

The input of the stereo matching algorithm is an image captured by a digital camera with different viewing angles, and the output is output. Is the correspondence of points on these images. Assuming a geometric model of binocular stereo vision in a standard configuration, c and c' are the optical centers of the two cameras, f is the focal length, and T is the line connecting the two optical centers, that is, the spacing between the two digital cameras, also called the baseline. A line that passes through the optical center and is perpendicular to the imaging plane is called the optical axis. The so-called standard configuration means that the optical axes of the two cameras are perpendicular to the baseline and parallel to each other. Let the focal lengths of the two cameras be equal to f, and the horizontal coordinate of the coordinate system of the camera is parallel to the baseline direction, then the point P in the space has the same vertical coordinate on the images formed by the two cameras. This feature is also called stereoscopic vision. The Epipolar Line (the so-called outer pole line refers to the intersection of the outer pole plane and the image plane, where the outer pole plane is a plane containing two focal points and spatial points). For cameras with a typical configuration, images can be obtained in standard configuration by camera calibration and registration. The image after P point projection to the two cameras is x and x', respectively, and x and x' are a pair of corresponding points. If x and x' are used to represent their horizontal coordinates, the correspondence between the two points can be described by the parallax defined as follows:

Parallax d=x-x'

By deriving a simple geometric relationship, we can get the following equation:

Where Z represents the depth of the corresponding point.

The terminal may obtain the disparity d of the boundary pixel points of the weak texture region using a stereo matching algorithm, and calculate the Z of the boundary pixel point of the weak texture region using the following formula:

Step 1304: Calculate a depth value of each pixel in the weak texture region according to the depth value of the boundary pixel.

The terminal may directly calculate the depth value of each pixel in the weak texture region according to the depth value of the acquired boundary pixel.

Due to the weakness of the stereo matching algorithm itself and the occlusion problem of the boundary point, a sudden pixel point with inaccurate partial depth values may occur. In this case, in order to obtain a more accurate depth value, the terminal may screen according to the depth value of the boundary pixel point. In addition to the abrupt point in the boundary pixel, a reliable point in the boundary pixel is obtained; the depth value of the reliable point is plane-fitted, and the depth value of each pixel in the weak texture region is calculated.

Optionally, the filtering the abrupt point in the boundary pixel according to the depth value of the boundary pixel includes: filtering the boundary pixel by using a RANSAC algorithm according to the depth value of the boundary pixel The point of mutation.

The RANSAC algorithm is an algorithm that calculates the mathematical model parameters of the data based on a set of sample data sets containing abnormal data and obtains valid sample data. The RANSAC algorithm is often used in computer vision.

The basic idea of the RANSAC algorithm is described as follows:

2 sets of samples in the SC=P\S and the model M whose error is less than a certain threshold t and S Form S*. S* is considered to be an inner set of points, which constitute a Consistus Set of S;

In this embodiment, the terminal may use the depth value of the boundary pixel as the sample set P, randomly extract the depth value of the n boundary pixel points from the sample set P as the subset S, and obtain the initialization model by plane fitting. M; dividing the depth value of the boundary pixel point in the residual set SC=P\S with the error of the initialization model M less than the second threshold into an inner point set; when the number of inner point concentration depth values reaches N, according to The inner point set recalculates the new model M* by least squares method; the new subset S* is re-randomly extracted, and the above process is repeated; after repeating a certain number of times (such as 5-10 times), the obtained largest inner point set is selected. a depth value of the maximum inner point set is a depth value of a reliable point in the boundary pixel point, and a remaining value in the sample set P is a depth value of a sudden change point in the boundary pixel point, the n sum N is the default value. n is a preset value, which can be 60%-80% of the number of sample points in P, and the N value is also a preset value, which can be 90% of the number of sample points in P.

The terminal may perform plane fitting on the depth value of the reliable point to obtain a plane fitting equation, and the depth value of each pixel in the weak texture region may be calculated by using the plane fitting equation. As shown in FIG. 10, a plane fitting diagram of the depth values of the boundary pixel points of a weak texture region, it can be seen that the plane of the boundary point fitting covers the weak texture region, and the depth values of other pixels in the region are It can be calculated by the plane fitting equation.

Those skilled in the art will appreciate that embodiments of the present invention can be provided as a method, system, or computer program product. Accordingly, the present invention can take the form of a hardware embodiment, a software embodiment, or a combination of software and hardware. Moreover, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) including computer usable program code.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

The above is only the preferred embodiment of the present invention and is not intended to limit the scope of the present invention.

Claims

A terminal comprising:

An image segmentation module configured to acquire color and brightness information of an image, and divide the image into a plurality of regions according to the color and brightness information;

a weak texture region acquiring module configured to calculate gradient information corresponding to the image, and select a weak texture region from the regions segmented by the image segmentation module according to the gradient information, where the weak texture region is a gradient statistical average An area within a preset range;

An edge depth obtaining module, configured to extract boundary pixel points of the weak texture region selected by the weak texture region acquiring module, and obtain a depth value of the boundary pixel point;

The area depth obtaining module is configured to calculate a depth value of each pixel point in the weak texture area according to the depth value of the boundary pixel point acquired by the edge depth acquiring module.
The method according to claim 1, wherein the weak texture region obtaining module is further configured to obtain gradient information corresponding to the image according to a gradient algorithm, where the gradient information is a gradient corresponding to each pixel in the image. Calculating a gradient statistical average corresponding to the pixel points in the plurality of image segmentation regions, and selecting a region in which the gradient statistical average value is within the preset range is a weak texture region.
The method according to claim 1, wherein the region depth obtaining module is further configured to: screen out a sudden change point in the boundary pixel point according to a depth value of the boundary pixel point, to obtain the boundary pixel point a reliable point; planarly fitting the depth value of the reliable point to calculate a depth value of each pixel in the weak texture region.
The terminal according to claim 1, wherein the image segmentation module is further configured to adopt a region-based segmentation method, or a threshold-based segmentation method, or an edge-based segmentation method, or a cluster analysis method, The image is divided into several regions.
The terminal according to claim 4, wherein the image segmentation module is further configured to select a plurality of seed pixels to map a new pixel around the seed pixel to meet a preset condition. Dividing into the region where the seed pixel is located, treating the new pixel point as a new seed pixel point, and continuing to divide a new pixel point satisfying the preset condition around the new seed pixel point to the seed pixel point An area in which the pixel points satisfying the preset condition are not present around the new seed pixel, and acquiring a plurality of regions according to the plurality of seed pixel points; wherein the preset condition is the seed The difference between the color and luminance information is within the first threshold compared to the pixel.
The terminal according to claim 1, wherein the edge depth obtaining module is further configured to perform area marking on the weak texture area to obtain a marked weak texture area, and then to mark the weak texture area and the image. The other regions are binarized to obtain the weakened texture region after binarization, and the binarized weak texture region is filled with the region to obtain the weakly textured region after the cavity is filled, and the weak texture after the cavity is filled The contour is extracted from the region to obtain the contour of the weak texture region, and the boundary pixel of the weak texture region is obtained according to the contour of the weak texture region; and the depth value Z of the boundary pixel is obtained according to the following formula:

Where f is the focal length of two digital cameras in the stereoscopic imaging device, T is the spacing between the two digital cameras, and d is the parallax corresponding to the boundary pixels in the disparity map of the two images taken by the two digital cameras value.
The terminal according to claim 6, wherein the area mark means: making the continuous area the same mark;

The edge depth obtaining module is further configured to perform a region marking on the weak texture region to obtain a marked weak texture region by using a four-neighbor labeling algorithm or an eight-neighbor labeling algorithm.
The terminal according to claim 6, wherein the edge depth obtaining module is further configured to adopt an area hole filling algorithm, and perform the hole filling after the binarized weak texture area to obtain the hole-filled weak texture area.
The terminal according to claim 1, wherein the edge depth obtaining module is further configured to apply a stereo matching algorithm to acquire a depth value of the boundary pixel.
The terminal according to claim 3, wherein the region depth obtaining module is further configured to extract the depth value of the n boundary pixel points from the sample set P by using the depth value of the boundary pixel as the sample set P. As the subset S, and obtaining the initialization model M by plane fitting; dividing the depth value of the boundary pixel point in the residual set SC=P\S with the error of the initialization model M smaller than the second threshold into the inner point set; When the number of depth values in the inner point reaches N, the new model M* is recalculated according to the inner point set by least squares method; the new subset S* is randomly selected, and the above process is repeated; after repeated a certain number of times, the selected one is selected. a obtained maximum inner point set, wherein the depth value in the maximum inner point set is a depth value of a reliable point in the boundary pixel point, and the remaining value in the sample set P is a depth of a sudden change point in the boundary pixel point Value, the n and N are preset values.
A method for acquiring depth information of a weak texture region in an image, the method comprising:

Obtaining color and brightness information of the image, and dividing the image into a plurality of regions according to the color and brightness information;

Obtaining gradient information corresponding to the image, and selecting a weak texture region from the plurality of regions according to the gradient information, where the weak texture region is an area in which a gradient statistical average value is within a preset range;

Extracting boundary pixel points of the weak texture region, and acquiring a depth value of the boundary pixel point;

Determining a depth value of each pixel in the weak texture region according to the depth value of the boundary pixel.
The method according to claim 11, wherein the calculating obtains gradient information corresponding to the image, and selecting a weak texture region from the plurality of regions according to the gradient information, comprising:

Obtaining gradient information corresponding to the image according to a gradient algorithm, where the gradient information is a gradient corresponding to each pixel point in the image;

Calculating a gradient statistical average corresponding to the pixel points in the plurality of image segmentation regions, and selecting a region in which the gradient statistical average value is within the preset range is a weak texture region.
The method according to claim 11, wherein the calculating the depth value of each pixel in the weak texture region according to the depth value of the boundary pixel point comprises:

And filtering a sudden change point in the boundary pixel point according to the depth value of the boundary pixel point to obtain a reliable point in the boundary pixel point; performing plane fitting on the depth value of the reliable point to calculate the weak point The depth value of each pixel in the texture area.
The method according to claim 11, wherein said dividing said image into a plurality of regions according to said color and brightness information comprises:

The image is segmented into a plurality of regions using a region-based segmentation method, a threshold-based segmentation method, or an edge-based segmentation method or a cluster analysis method.
The method of claim 14, wherein the segmenting the image into a plurality of regions using a region-based segmentation method comprises:

Selecting a plurality of seed pixels, dividing a new pixel point satisfying the preset condition around the seed pixel point into an area where the seed pixel point is located, and treating the new pixel point as a new seed pixel point, and continuing to A new pixel point that satisfies a preset condition around the new seed pixel is divided into an area where the seed pixel point is located, until there is no pixel point satisfying the preset condition around the new seed pixel point, and the obtained a plurality of regions divided by the seed pixel; wherein the preset condition is that the difference between the color and the luminance information is within the first threshold compared to the seed pixel.
The method of claim 11, wherein the extracting the boundary pixel points of the weak texture region and obtaining the depth value of the boundary pixel points comprises:

Performing area marking on the weak texture area to obtain the marked weak texture area, and then binarizing the marked weak texture area and other areas of the image to obtain a binarized weak texture area, and the second The weakened texture region is filled with the region to obtain the weakly textured region after the cavity is filled, and the weakly textured region after the cavity is filled for contour extraction to obtain the contour of the weakly textured region, according to the weak texture region The contour line acquires the weak texture region a boundary pixel; the depth value Z of the boundary pixel is obtained according to the following formula:

Where f is the focal length of two digital cameras in the stereoscopic imaging device, T is the spacing between the two digital cameras, and d is the parallax corresponding to the boundary pixels in the disparity map of the two images taken by the two digital cameras value.
The method according to claim 16, wherein said area mark means: making the continuous area the same mark;

And performing the area marking on the weak texture area to obtain the marked weak texture area, including:

The weak texture region is region-marked to obtain the marked weak texture region by using a four-neighbor labeling algorithm or an eight-neighbor labeling algorithm.
The method according to claim 16, wherein the performing the hole filling of the binarized weak texture region to obtain the void-filled weak texture region comprises:

The area void filling algorithm is used to fill the weakly textured area of the binarized area to obtain the weakly textured area after the cavity is filled.
The method of claim 11, wherein the obtaining the depth value of the boundary pixel comprises:

A stereo matching algorithm is applied to obtain a depth value of the boundary pixel.
The method of claim 13, wherein the filtering the abrupt points in the boundary pixel points according to the depth value of the boundary pixel points to obtain a reliable point in the boundary pixel points comprises:

Taking the depth value of the boundary pixel as the sample set P, randomly extracting the depth values of the n boundary pixel points from the sample set P as the subset S, and obtaining the initialization model M by plane fitting; the residual set SC=P The depth value of the boundary pixel point in the \S with the initialization model M whose error is less than the second threshold is divided into inner point sets; when the number of inner point concentration depth values reaches N, the least square method is used according to the inner point set. Calculate the new model M*; re-randomly extract the new subset S*, heavy Repeating the above process; after repeating a certain number of times, selecting the obtained maximum inner point set, the depth value of the maximum inner point set is a depth value of the reliable point in the boundary pixel point, and the rest of the sample set P The value is the depth value of the abrupt point in the boundary pixel, and the n and N are preset values.