WO2016095081A1 - Recherche de ligne de visée à l'aide des informations de balayage de trame - Google Patents

Recherche de ligne de visée à l'aide des informations de balayage de trame Download PDF

Info

Publication number
WO2016095081A1
WO2016095081A1 PCT/CN2014/093815 CN2014093815W WO2016095081A1 WO 2016095081 A1 WO2016095081 A1 WO 2016095081A1 CN 2014093815 W CN2014093815 W CN 2014093815W WO 2016095081 A1 WO2016095081 A1 WO 2016095081A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
sample
region
map
pixel
Prior art date
Application number
PCT/CN2014/093815
Other languages
English (en)
Inventor
Zhijie Deng
Lei Zhang
Rahul SURESH
Daniel LUKA
Original Assignee
Qualcomm Incorporated
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qualcomm Incorporated filed Critical Qualcomm Incorporated
Priority to PCT/CN2014/093815 priority Critical patent/WO2016095081A1/fr
Publication of WO2016095081A1 publication Critical patent/WO2016095081A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation

Definitions

  • This disclosure is generally related to image processing, such as line-of-sight search operations.
  • An electronic device may include a processor and a camera that captures images.
  • the processor may perform image processing operations, such as by modifying image data that represents an image.
  • image processing operation may include separating (e.g., segmenting) a foreground portion and a background portion of an image (e.g., so that the background portion can be modified or removed from the image) .
  • Image matting operations may be performed to improve accuracy of image processing operations. For example, a boundary portion between a foreground portion and a background portion may be searched for a sample of the foreground portion. If one or more pixels of the boundary portion “match” the sample, then the one or more pixels may be associated with the foreground portion.
  • Image matting operations may include line-of-sight (LoS) searches of image data (e.g., by searching a search area of the image data along a straight line) .
  • Search areas for LoS searches are typically selected randomly or pseudo-randomly.
  • a cache storing the image data may be accessed irregularly or non-sequentially, which may result in cache misses and poor performance.
  • attempting to select search areas for LoS searches deterministically (e.g., sequentially, in order to reduce cache misses) may result in poor image quality in some cases (e.g., due to image artifacts) .
  • a device in accordance with the disclosure may generate information related to image data prior to performing one or more line-of-sight (LoS) searches of the image data.
  • the device may access the information to determine whether to perform a particular LoS search and/or to determine a start pixel of a LoS search. For example, if the information indicates that a search area corresponding to a particular LoS search does not include foreground (or background) samples of the image data, then the device may select another LoS search that includes foreground (or background) samples. As another example, if information indicates that the search area corresponding to a LoS search includes one or more “gaps” that do not include foreground (or background) samples, the device may “step” over the gaps based on the information.
  • LoS line-of-sight
  • the device may perform raster scans of the image data to generate the information. Because raster scans may be performed in an ordered manner (e.g., from top-left to bottom-right relative to an image) , the raster scans may not substantially increase computational complexity associated with image processing operations. Further, the information can be “shared” between multiple LoS searches (e.g., by using the information to determine whether to perform, and where to begin, each of multiple LoS searches) . As a result, inefficient LoS searches can be avoided, such as LoS searches of search areas that do not include foreground (or background) samples. Because LoS searches are associated with high computational complexity and cache misses, reducing a number of LoS searches may improve device performance. Further, efficiency can be increased by “stepping” over gaps in a search area (e.g., by initializing a LoS search at a foreground (or background) sample instead of beginning at a randomly selected pixel) .
  • a method of image processing performed by a device includes identifying a first region of an image, a second region of the image, and a third region of the image. The method further includes generating information related to the image using raster scans of image data corresponding to the image. Based on the information, a search area is selected for a line-of-sight search of the image data.
  • an apparatus in another particular example, includes a segmenting module configured to identify a first region of an image, a second region of the image, and a third region of the image.
  • the apparatus further includes a raster scanner configured to generate information related to the image using raster scans of image data corresponding to the image.
  • the apparatus further includes a selector configured to select, based on the information, a search area for a line-of-sight search of the image data.
  • an apparatus in another particular example, includes means for identifying a first region of an image, a second region of the image, and a third region of the image.
  • the apparatus further includes means for generating information related to the image using raster scans of image data corresponding to the image.
  • the apparatus further includes means for selecting, based on the information, a search area for a line-of-sight search of the image data.
  • a computer-readable medium stores instructions executable by a processor to perform operations during image processing of image data of an image.
  • the operations include identifying a first region of the image, a second region of the image, and a third region of the image.
  • the operations further include generating information related to the image using raster scans of the image data.
  • the operations further include selecting, based on the information, a search area for a line-of-sight search of the image data.
  • One particular advantage provided by at least one of the disclosed embodiments is a reduction in use of computational resources associated with image processing operations.
  • low-complexity raster scans may be performed to generate pre-processing information, which may be used to simplify a line-of-sight search (e.g., by avoiding “gaps” in a line-of-sight search) and/or to avoid non-useful line-of-sight searches (e.g., line-of-sight searches that do not include samples of interest) .
  • line-of-sight search e.g., by avoiding “gaps” in a line-of-sight search
  • non-useful line-of-sight searches e.g., line-of-sight searches that do not include samples of interest
  • FIG. 1 is a diagram that depicts certain illustrative aspects of a device that may perform line-of-sight searching of image data based on information generated using raster scans of the image data.
  • FIG. 2 illustrates a map, such as a foreground sample availability map, that may be represented by the information generated by the device of FIG. 1.
  • FIG. 3 illustrates a map, such as a background sample availability map, that may be represented by the information generated by the device of FIG. 1.
  • FIG. 4 illustrates a map, such as a foreground sample clearance map, that may be represented by the information generated by the device of FIG. 1.
  • FIG. 5 illustrates a map, such as a background sample clearance map, that may be represented by the information generated by the device of FIG. 1.
  • FIG. 6 is a flow diagram that depicts an illustrative method of operation of a device, such as the device of FIG. 1.
  • FIG. 7 is a block diagram of an illustrative example of an electronic device that may perform line-of-sight searching of image data based on information generated using raster scans of the image data.
  • FIG. 1 depicts certain illustrative aspects of a device 100.
  • the device 100 may be integrated within a mobile device (e.g., a cellular telephone) , a computer (e.g., a laptop computer, a tablet computer, or a desktop computer) , a set top box, an entertainment unit, a navigation device, a personal digital assistant (PDA) , a television, a tuner, a radio (e.g., a satellite radio) , a music player (e.g., a digital music player and/or a portable music player) , a video player (e.g., a digital video player, such as a digital video disc (DVD) player and/or a portable digital video player) , an automotive system console, a home appliance, a wearable device (e.g., a personal camera, a head mounted display, and/or a watch) , a robot, a healthcare device, or another electronic device, as illustrative examples.
  • the device 100 may include one or more cameras, such as a camera 104.
  • the camera 104 may include a charge-coupled device (CCD) or a complementary-oxide-metal-semiconductor (CMOS) device, as illustrative examples.
  • the device 100 may further include a segmenting module 108, a sampler 114, a raster scanner 120, a selector 124, and a line-of-sight (LoS) searcher 128.
  • CCD charge-coupled device
  • CMOS complementary-oxide-metal-semiconductor
  • the device 100 may capture one or more images of a scene 102.
  • the camera 104 may capture an image of the scene 102 to generate image data 106 representing the scene 102.
  • the image data 106 may be stored at a memory, such as a buffer or at a cache of the device 100, as illustrative examples.
  • the device 100 may be configured to perform image processing operations using the image data 106.
  • the device 100 may be configured to perform computer vision image processing operations using the image data 106, such as to track and/or recognize an object 103 within the scene 102, as illustrative examples.
  • the image processing operations may include image segmenting operations.
  • the segmenting module 108 may be configured to segment the image data 106 into sets of image data representing multiple corresponding regions.
  • the multiple regions may include a first region 110a (e.g., a foreground region, such as a region including the object 103) , a second region 110b (e.g., a background region) , and a third region 110c (e.g., an “undecided” region that may include foreground and/or background portions) .
  • the multiple regions include three regions.
  • the segmenting module 108 may segment the image data 106 into fewer than three regions (e.g., two regions) or into more than three regions (e.g., with multiple foreground regions, multiple background regions, and/or multiple undecided regions) .
  • the device 100 may be configured to associate the third region 110c with either the first region 110a or the second region 110b based on a search of the image data 106.
  • the device 100 may perform a line-of-sight (LoS) search of the image data 106, such as by generating a sample 112 of the first region 110a using the sampler 114 and by searching the third region 110c for areas that are similar to the sample 112 (e.g., having a similar color, as an illustrative example) .
  • the device 100 may perform an LoS search of the image data 106 by obtaining a sample of the second region 110b and searching the third region 110c for areas that are similar to the sample of the second region 110b.
  • the device 100 is configured to use raster scans of the image data 106 to simply an LoS search process.
  • the device 100 is configured to partition the image data 106 into multiple regions, and the raster scanner 120 is configured to perform multiple raster scans based on a number of the multiple regions to generate information 122.
  • the device 100 may utilize the information 122 in connection with one or more LoS searches of the image data 106, such as by using the information 122 to determine one or more particular areas of the image data 106 to search.
  • the information 122 may include pre-processing information that is generated prior to LoS searches of the image data 106 and that is usable in connection with the LoS searches (e.g., that can be “re-used” or “shared” between multiple LoS searches) .
  • FIG. 1 depicts a partitioning scheme 150.
  • the device 100 may partition the image data 106 into multiple sectors (sectors 0, 1, 2, 3, 4, 5, 6, and 7) based on the partitioning scheme 150.
  • the image data 106 is segmented into sectors of the image data 106, and raster scans may be performed to determine information regarding each of the sectors.
  • each of sectors is associated with a 45 degree range.
  • a different partitioning scheme may be used, such as by partitioning the image data 106 into four or sixteen sectors, as illustrative examples.
  • a raster scan 160 may be performed over the image data 106. For example, if the image data 106 is associated with rows and columns, the raster scan 160 may be performed from the “top-left” pixel (or the pixel in the first row and first column) to the “top-right” pixel (or the pixel in the first row and last column) , and then for the second row onward until reaching the “bottom-right” pixel (or the pixel in the last row and last column) . The raster scan 160 may scan each pixel of the image data 106 for pixels that are similar to (or the same as) the sample 112.
  • the device 100 may use the raster scan 160 for sector 0 to determine, for each pixel of the image data 106, whether neighbor pixels (within sector 0) of the pixel are foreground (or background) samples.
  • the device 100 may use the raster scan 160 to determine whether a neighbor pixel “A” is a foreground sample, whether a neighbor pixel “B” is a foreground sample, whether sector 0 includes any foreground samples “beyond” (excluding) neighbor pixel “A” within a range 164, and whether sector 0 includes any foreground samples “beyond” (excluding) pixel “B” within a range 166.
  • the device 100 may determine that sector 0 includes at least one foreground sample pixel beyond (excluding) the selected pixel 162.
  • the device 100 is configured to modify a 0th bit of a data structure hasFG to indicate that sector 0 includes at least one foreground sample pixel beyond (excluding) the selected pixel 162.
  • the device 100 may determine that sector 0 does not include at least one foreground sample pixel beyond (excluding) the selected pixel 162.
  • the device 100 is configured to modify the 0th bit of the data structure hasFG to indicate that sector 0 does not include at least one foreground sample pixel beyond (excluding) the selected pixel 162.
  • the raster scan 160 may be performed for each pixel of the image data 106 to determine if neighbor regions of each pixel within sector 0 include one or more foreground samples. Alternatively or in addition, a raster scan may be performed for each pixel of the image data 106 to determine if neighbor regions of each pixel within sector 0 include one or more background samples.
  • the device 100 may be configured to modify a 0th bit of a data structure hasBG to indicate that sector 0 does not include at least one background sample.
  • raster scans are performed for each of the sectors 0, 1, 2, 3, 4,5, 6, and 7.
  • the device 100 may use raster scans to determine values for bits 0, 1, 2, 3, 4, 5, 6, and 7 for the data structure hasFG (for sectors 0, 1, 2, 3, 4, 5, 6, and 7, respectively) and to determine values for bits 0, 1, 2, 3, 4, 5, 6, and 7 for the data structure hasBG (for sectors 0, 1, 2, 3, 4, 5, 6, and 7, respectively) .
  • Table 1 illustrates logic corresponding to an example mapping of bit values of the data structure hasFG.
  • Table 2 illustrates logic corresponding to an example mapping of bit values of the data structure hasBG:
  • the data structures hasFG and hasBG are computed for each pixel of the image data 106.
  • the information 122 of FIG. 1 may indicate the bit values of the data structures hasFG and hasBG.
  • a direction (or order) of a raster scan may be selected so that “children” (or neighbors) of a selected pixel are scanned before the selected pixel is scanned.
  • the raster scan 160 scans from left to right and from top to bottom with respect to the image data 106 (i.e., from top-left to bottom-right) , pixels “A” and “B” are scanned before the selected pixel 162.
  • a raster scan from top-left to bottom-right may be performed to determine bit values of the data structures hasFG and hasBG for sector 0 and to determine bit values of the data structures hasFG and hasBG for sector 1.
  • a raster scan from bottom-left to top-right may be performed to determine bit values of the data structures hasFG and hasBG for sector 2 and to determine bit values of the data structures hasFG and hasBG for sector 3.
  • a raster scan from bottom-right to top-left may be performed to determine bit values of the data structures hasFG and hasBG for sector 4 and to determine bit values of the data structures hasFG and hasBG for sector 5.
  • a raster scan from top-right to bottom-left may be performed to determine bit values of the data structures hasFG and hasBG for sector 6 and to determine bit values of the data structures hasFG and hasBG for sector 7.
  • the device 100 may perform four raster scans to determine bit values of the data structures hasFG and hasBG for each of eight sectors. In other implementations, a different number of raster scans and/or sectors may be used.
  • the device 100 is further configured to determine values of a data structure FGstep and values of a data structure BGstep.
  • the data structure FGstep may indicate, for a particular pixel of the image data 106, a distance from the particular pixel to a nearest foreground sample.
  • the data structure FGstep may indicate a radius number of pixels (extending from the particular pixel, excluding the particular pixel and any perimeter pixels) for which the image data 106 does not include one or more foreground samples.
  • the data structure FGstep may be computed for each pixel of the image data 106.
  • the data structure BGstep may indicate, for a particular pixel of the image data 106, a distance from the particular pixel to a nearest background sample.
  • the data structure BGstep may indicate a radius number of pixels (extending from the particular pixel, excluding the particular pixel and any perimeter pixels) for which the image data 106 does not include one or more background samples.
  • the data structure BGstep may be computed for each pixel of the image data 106.
  • the information 122 of FIG. 1 may indicate the data structures FGstep and BGstep.
  • Table 3 illustrates pseudo-code corresponding to an example of a process that may be performed at the device 100.
  • a process in accordance with Table 3 may include defining ( “typedef” ) a data structure ( “struct” ) for each pixel of the image data 106.
  • Each data structure may include a first field, such as an unsigned 8-bit integer ( “unint8_t” ) that indicates bit values ofhasFG.
  • Each data structure may further include a second field, such as another unsigned 8-bit integer that indicates bit values ofhasBG.
  • Each data structure may also include a third field, such as an unsigned 16-bit integer (uint16_t) that indicates bit values of FGstep.
  • Each data structure may further include a fourth field, such as an unsigned 16-bit integer that indicates bit values of BGstep.
  • the process may further include generating one or more maps using the first field, the second field, the third field, and the fourth field ( “gather_map_t” ) . Illustrative examples of maps are described further with reference to FIGS. 2-5.
  • the device 100 may use the information 122 in connection with one or more LoS searches of the image data 106 (e.g., in order to determine whether portions of the third region 110c should be assigned to the first region 110a or to the second region 110b) .
  • FIG. 1 depicts an example of an image 170 that may correspond to the image data 106.
  • the image 170 depicts foreground region 172 (e.g., a figure, which may correspond to the object 103) , a background region 174, and an undecided region 176.
  • the undecided region 176 may include a boundary between the foreground region 172 and the background region 174) .
  • the foreground region 172 may correspond to the first region 110a
  • the background region 174 may correspond to the second region 110b
  • the undecided region 176 may correspond to the third region 110c.
  • the undecided region 176 may include an edge region of the image and/or another region, such as a region within the foreground region 172 and/or a region within the background region 174.
  • the LoS searcher 128 may determine (e.g., randomly or pseudo-randomly determine) lines-of-sight along which searches are to be performed for pixels similar to a sample (e.g., to search for pixels similar to the sample 112) .
  • the image 170 depicts an LoS 178 that may be randomly or pseudo-randomly determined by the LoS searcher 128, such as by selecting a pixel and a trajectory within the image 170.
  • the selector 124 may access the information 122 to determine whether the LoS 178 includes one or more foreground pixels (e.g., to determine whether a foreground search is to be performed along the LoS 178) . For example, the selector 124 may determine a particular sector of the sectors 0-7 to which the LoS 178 belongs (e.g., sector 5) and may access the hasFG bit value for the selected pixel and the particular sector to determine whether the LoS 178 includes one or more foreground pixels. If the LoS 178 includes one or more foreground pixels, the selector 124 may cause the LoS searcher 128 to search the LoS 178 (e.g., by accessing a cache of the device 100 to retrieve pixel data corresponding to the LoS 178) . If the LoS 178 does not include one or more foreground pixels, the selector 124 may cause the LoS searcher 128 to determine one or more other lines-of-sight to be searched, such as an LoS 179.
  • the selector 124 may access the information 122 to determine whether the LoS 178 includes one or more background pixels (e.g., to determine whether a background search is to be performed along the LoS 178) . As an example, the selector 124 may access the hasBG bit value for the selected pixel and the particular sector associated with the LoS 178. If the LoS 178 includes one or more background pixels, the selector 124 may cause the LoS searcher 128 to search the LoS 178 (e.g., by accessing a cache of the device 100 to retrieve pixel data corresponding to the LoS 178) . If the LoS 178 does not include one or more background pixels, the selector 124 may cause the LoS searcher 128 to determine one or more other lines-of-sight to be searched, such as the LoS 179.
  • the selector 124 may determine one or more “jumps” along the LoS. For example, if the selector 124 determines (e.g., based on the hasFG data structure) that a foreground search of the LoS 178 is to be performed, the selector 124 may access the FGstep data structure to determine a number of pixels along the LoS 178 from the selected pixel that do not include a foreground pixel (i.e., a number of pixels to “jump” from the selected pixel to arrive at a foreground pixel) .
  • the selector 124 may provide an indication of a pixel to which the LoS searcher 128 is to “jump” to begin the LoS 178.
  • the selector 124 may access the BGstep data structure to determine a number of pixels along the LoS 178 from the selected pixel that do not include a background pixel (i.e., a number of pixels to “jump” from the selected pixel to arrive at a background pixel) .
  • the selector 124 may provide an indication of a pixel to which the LoS searcher 128 is to “jump” to begin the LoS 178.
  • the device 100 may assign one or more portions of the third region 110c to the first region 110a and/or to the second region 110b. For example, the device 100 may determine, using one or more LoS search operations, that one or more portions of the third region 110c are included in the first region 110a (e.g., a foreground region) and/or that one or more portions of the third region 110c are included in the second region 110b (e.g., a background region) .
  • the first region 110a e.g., a foreground region
  • the second region 110b e.g., a background region
  • the device 100 may assign one or more portions of the third region 110c to the first region 110a and/or the device 100 may assign one or more portions of the third region 110c to the second region 110b, resulting in matted image data 130.
  • the matted image data 130 may be used in one or more image processing operations, such as a computer vision application, as an illustrative example.
  • the techniques of FIG. 1 may reduce complexity of image matting operations. For example, because raster scans may use relatively simple cache accesses (e.g., “regular” or sequential cache accesses) and LoS searches may use more complex cache accesses (e.g., irregular or random cache accesses, which may result in cache misses) , use of raster scans to reduce a number of LoS searches (or to assist with LoS searches) may reduce time and complexity associated with image matting operations.
  • raster scans may use relatively simple cache accesses (e.g., “regular” or sequential cache accesses) and LoS searches may use more complex cache accesses (e.g., irregular or random cache accesses, which may result in cache misses)
  • use of raster scans to reduce a number of LoS searches may reduce time and complexity associated with image matting operations.
  • FIG. 2 illustrates an example of a map 200, such as a foreground sample availability map.
  • the information 122 of FIG. 1 indicates (e.g., includes data representing) the map 200.
  • the map 200 may be generated by determining values of the data structure hasFG for each pixel of the image data 106.
  • each pixel of the map 200 may correspond to a data structure hasFG for a corresponding pixel of the image data 106.
  • each data structure hasFG includes one byte (e.g., one bit for each of the sectors 0-7) .
  • a data structure hasFG for a particular pixel may have a value of between 0 and 255, and the example of FIG. 2 depicts a legend 250 with values ranging from 0 to 255.
  • a value of 0 may indicate that no sectors adjacent to the particular pixel include one or more foreground samples
  • a value of 255 may indicate that all sectors adjacent to the particular pixel include one or more foreground samples.
  • an area 202 of the map 200 may be associated with larger hasFG values (e.g., of between 240 and 255, indicating that each pixel of the area 202 is associated with a relatively large number of neighboring foreground samples) .
  • an area 204 of the map 200 may be associated with lower hasFG values (e.g., of between 0 and 50, indicating that each pixel of the area 204 is associated with few or zero adjacent neighboring samples) .
  • the map 200 may indicate, for each pixel of the image data 106, availability of neighboring foreground samples (e.g., for each of the sectors 0-7) .
  • the map 200 may be accessed by the selector 124 of FIG. 1 to identify lines-of-sight that include one or more foreground samples.
  • an LoS that extends through the area 202 may include a large number of foreground samples, and the LoS may be selected for a LoS search.
  • the map 200 may be utilized by the selector 124 to identify lines-of-sight that do not include foreground samples.
  • an LoS of a search area extending through the area 204 may have few or zero foreground samples, and the LoS search may be avoided (e.g., in favor of another LoS that includes one or more foreground samples) .
  • FIG. 3 illustrates an example of a map 300, such as a background sample availability map.
  • the information 122 of FIG. 1 indicates (e.g., includes data representing) the map 300.
  • the map 300 may be generated by determining values of the data structure hasBG for each pixel of the image data 106.
  • each pixel of the map 300 may correspond to a data structure hasBG for a corresponding pixel of the image data 106.
  • each data structure hasBG includes one byte (e.g., one bit for each of the sectors 0-7) .
  • a data structure hasBG for a particular pixel may have a value of between 0 and 255, and the example of FIG. 3 illustrates a legend 350 with values ranging from 0 to 255.
  • a value of 0 may indicate that no sectors adjacent to the particular pixel include one or more background samples
  • a value of 255 may indicate that all sectors adjacent to the particular pixel include one or more background samples.
  • an area 302 of the map 300 may be associated with larger hasBG values (e.g., of greater than 200, indicating that each pixel of the area 302 associated with a relatively large number of neighboring background samples) .
  • an area 304 of the map 300 may be associated with lower hasBG values (e.g., of between 0 and 50, indicating that each pixel of the area 304 is associated with few or zero neighboring background samples) .
  • the map 200 may indicate, for each pixel of the image data 106, availability of neighboring foreground samples (e.g., for each of the sectors 0-7) .
  • the map 300 may be accessed by the selector 124 of FIG. 1 to identify lines-of-sight that include one or more background samples.
  • an LoS that extends through the area 302 may include a large number of background samples, and the LoS may be selected for a LoS search.
  • the map 300 may be utilized by the selector 124 to identify lines-of-sight that do not include background samples.
  • an LoS extending through the area 304 may have fewer background samples than an LoS extending through the area 302.
  • the LoS search of a search area extending through the area 304 may be avoided (e.g., in favor of another LoS that includes a greater number of background samples) .
  • FIG. 4 illustrates an example of a map 400, such as a foreground sample clearance map.
  • the information 122 of FIG. 1 indicates (e.g., includes data representing) the map 400.
  • the map 400 may be generated by determining values of the data structure FGstep for each pixel of the image data 106.
  • each pixel of the map 400 may correspond to a data structure FGstep for a corresponding pixel of the image data 106.
  • the data structure FGstep may indicate, for a particular pixel of the image data 106, a number of pixels that do not include one or more foreground samples (e.g., a radius from the particular pixel to the nearest foreground sample) .
  • a data size (e.g., number of bits) of the data structure FGstep may be determined based on the particular application. In FIGS.
  • each map includes approximately 1800 pixel columns and approximately 1000 pixel rows, resulting in approximately 1,800,000 pixels
  • values of the data structure FGstep may range from 0 to approximately 2060 (because sqrt (1800 ⁇ 2 +1000 ⁇ 2) ⁇ 2060) .
  • the data structure FGstep may include eleven bits (for 2056 combinations of values) in order to approximately represent each possible number of pixels in a particular direction that do not include one or more foreground samples.
  • the data structure FGstep may include fewer bits than needed to represent the largest possible number of pixels (e.g., to reduce computational complexity)
  • a legend 450 indicates that values of the data structure FGstep may range from 0 to 350, as an illustrative example.
  • an area 402 of the map 400 may be associated with smaller FGstep values (e.g., zero, which indicates that each pixel of the area 402 has a neighboring foreground sample) .
  • an area 404 of the map 400 may be associated with greater FGstep values (e.g., of approximately 350, indicating that, for each pixel of the area 404, approximately 350 non-foreground samples separate the pixel and a nearest foreground sample) .
  • the map 400 may indicate, for each pixel of the image data 106, foreground “clearance” of the pixel (e.g., a distance from the pixel to a nearest foreground sample) .
  • the map 400 may be used to perform multiple jumps in connection with a foreground search (e.g., to avoid multiple “gaps” in the image data that 106 do not include foreground samples) .
  • the map 400 may be accessed to determine whether to “jump” from a starting pixel of a LoS for a LoS search.
  • the selector 124 of FIG. 1 selects a particular LoS for a LoS search that begins at a selected pixel
  • the device 100 may access the map 400 to determine whether to “jump” from the selected pixel to another pixel.
  • the LoS begins in the area 402
  • the device 100 may determine that no jump (or a jump of zero pixels) is to be performed for a foreground search (because each pixel of the area 402 is associated with a neighboring foreground sample) .
  • the device 100 may determine that a jump is to be performed for a foreground search (e.g., by jumping approximately 350 pixels to the nearest foreground sample) . Accordingly, a number of computations associated with an LoS search may be reduced by using the map 400 to “jump” past pixels that do not include foreground samples (or that include less than a threshold number of foreground samples) .
  • FIG. 5 illustrates an example of a map 500, such as a background sample clearance map.
  • the information 122 of FIG. 1 indicates (e.g., includes data representing) the map 500.
  • the map 500 may be generated by determining values of the data structure BGstep for each pixel of the image data 106.
  • each pixel of the map 500 may correspond to a data structure BGstep for a corresponding pixel of the image data 106.
  • the data structure BGstep may indicate, for a particular pixel of the image data 106, a number of pixels that do not include one or more background samples (e.g., a radius from the particular pixel to the nearest background sample) .
  • a data size (e.g., number of bits) of the data structure BGstep may be determined based on the particular application.
  • a legend 550 indicates that values of the data structure BGstep may range from 0 to 350, as an illustrative example.
  • an area 502 of the map 500 may be associated with smaller BGstep values (e.g., zero, which indicates that each pixel of the area 502 has a neighboring background sample) .
  • an area 504 of the map 500 may be associated with greater BGstep values (e.g., of approximately 200, indicating that, for each pixel of the area 504, approximately 200 non-background samples separate the pixel and a nearest background sample) .
  • the map 500 may indicate, for each pixel of the image data 106, background “clearance” of the pixel (e.g., a distance from the pixel to a nearest background sample) .
  • the map 500 may be used to perform multiple jumps in connection with a background search (e.g., to avoid multiple “gaps” in the image data that 106 do not include background samples) .
  • the map 500 may be accessed to determine whether to “jump” from a starting pixel of a LoS for a LoS search.
  • the selector 124 of FIG. 1 selects a particular LoS for a LoS search that begins at a selected pixel
  • the device 100 may access the map 500 to determine whether to “jump” from the selected pixel to another pixel.
  • the LoS begins in the area 502
  • the device 100 may determine that no jump (or a jump of zero pixels) is to be performed for a background search (because each pixel of the area 502 is associated with a neighboring background sample) .
  • the device 100 may determine that a jump is to be performed for a background search (e.g., by jumping approximately 200 pixels to the nearest background sample) . Accordingly, a number of computations associated with an LoS search may be reduced by using the map 500 to “jump” past pixels that do not include foreground samples (or that include less than a threshold number of foreground samples) .
  • FIG. 6 a particular illustrative embodiment of a method of image processing performed by a device is depicted and generally designated 600.
  • the device may correspond to the device 100 of FIG. 1, and the image processing may include image matting operations.
  • the method 600 may include identifying a first region of an image, a second region of the image, and a third region of the image, at 602.
  • the first region may correspond to the first region 110a
  • the second region may correspond to the second region 110b
  • the third region may correspond to the third region 110c.
  • the first region may correspond to a foreground of the image (e.g., the foreground region 172)
  • the second region may correspond to a background of the image (e.g., the background region 174)
  • the third region may correspond to a portion of the image that includes the background and/or the foreground of the image (e.g., the undecided region 176)
  • the third region may include a boundary region between the first region and the second region and/or a region between the first or second regions and an edge of the image.
  • the method 600 may further include generating information related to the image using raster scans of image data corresponding to the image, at 604.
  • the information may correspond to the information 122
  • the image data may correspond to the image data 106
  • the raster scans may include the raster scan 160 described with reference to FIG. 1.
  • generating the information 122 may include partitioning the image data 106 into multiple sectors, such as the sectors 0-7 described with reference to FIG. 1, and generating information for each of the multiple sectors.
  • the method 600 may further include selecting a search area for a line-of-sight search of the image data, at 604.
  • the search area may include pixels associated with an LoS, such as the LoS 178 or the LoS 179.
  • the search area is selected based on the information.
  • the information 122 may represent a map (e.g., the map 200) that indicates, for each pixel of the image, a number of neighboring pixels associated with a sample of the image (e.g., the sample 112) targeted by the line-of-sight search,
  • the search area may be selected based on a determination that the search area includes one or more foreground samples.
  • the sample of the image may be a sample of a foreground of the image (e.g., the foreground region 172)
  • the map may be a foreground sample availability map.
  • the information 122 may represent another map, such as the map 300.
  • the sample may be a sample of a background of the image (e.g., the background region 174)
  • the map may be a background sample availability map (e.g., the map 300) .
  • the information 122 may represent a map that indicates, for each pixel of the image, a distance from the pixel to another pixel that is associated with a sample of the image (e.g., the sample 112) targeted by the line-of-sight search.
  • the sample may be a sample of a foreground of the image (e.g., the foreground region 172)
  • the sample availability map may be a foreground sample clearance map (e.g., the map 400) .
  • the information may represent another map, such as the map 500.
  • the sample may be a sample of a background of the image (e.g., the background region 174)
  • the sample availability map may be a background sample clearance map (e.g., the map 500)
  • the method 600 may further include adjusting a start pixel associated with the line-of-sight search based on the information (e.g., by “jumping” from a start pixel associated with a foreground search to a foreground sample based on the map 400 and/or by “jumping” from a start pixel associated with a background search to a background sample based on the map 500) .
  • the method 600 may further include performing the line-of-sight search (e.g., by searching the search area for one or more pixels corresponding to the sample 112) and determining, based on a result of the line-of-sight search, that one or more pixels of the third region are associated with either the foreground or the background. For example, if a foreground search of the search area identifies one or more pixels corresponding to a foreground sample, then the one or more pixels may be associated with the first region 110a (e.g., the one or more pixels may be “reassigned” from the third region 110c to the first region 110a) . In this example, the method 600 may further include determining, based on the information, that the search area includes one or more foreground samples.
  • the method 600 may further include determining, based on the information, that the search area includes one or more background samples.
  • the method 600 may enable simplified image processing operations. For example, by determining whether (and where) to begin LoS searches, complexity of image matting operations may be reduced (e.g., by reducing a number of irregular cache accesses at a cache that stores image data, thereby reducing a number of caches misses at the cache) .
  • One or more operations of the method 600 may be initiated, controlled, or performed by a hardware device, such as a processing unit.
  • a hardware device such as a processing unit.
  • operations of the method 600 may be initiated, controlled, or performed by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC) , a processing unit such as a central processing unit (CPU) , a digital signal processor (DSP) , a graphics processing unit (GPU) , a controller, another hardware device, a firmware device, or a combination thereof.
  • FPGA field-programmable gate array
  • ASIC application-specific integrated circuit
  • processing unit such as a central processing unit (CPU) , a digital signal processor (DSP) , a graphics processing unit (GPU) , a controller, another hardware device, a firmware device, or a combination thereof.
  • CPU central processing unit
  • DSP digital signal processor
  • GPU graphics processing unit
  • controller another hardware device, a firmware device, or a combination
  • FIG. 7 a block diagram of a particular illustrative embodiment of an electronic device is depicted and generally designated 700.
  • One or more structures and operations of the electronic device 700 may correspond to the device 100 of FIG. 1 (e.g., the device 100 may be integrated within the electronic device 700) .
  • the electronic device 700 may correspond to a mobile device (e.g., a cellular telephone) , a computer (e.g., a laptop computer, a tablet computer, or a desktop computer) , a set top box, an entertainment unit, a navigation device, a personal digital assistant (PDA) , a television, a tuner, a radio (e.g., a satellite radio) , a music player (e.g., a digital music player and/or a portable music player) , a video player (e.g., a digital video player, such as a digital video disc (DVD) player and/or a portable digital video player) , an automotive system console, a home appliance, a wearable device (e.g., a personal camera, a head mounted display, and/or a watch) , a robot, a healthcare device, another electronic device, or a combination thereof.
  • a mobile device e.g., a cellular telephone
  • a computer e.
  • the electronic device 700 may include one or more processors, such as a processor 710.
  • the processor 710 may include a digital signal processor (DSP) , a central processing unit (CPU) , and/or a graphics processing unit (GPU) , as illustrative examples.
  • DSP digital signal processor
  • CPU central processing unit
  • GPU graphics processing unit
  • the processor 710 may utilize data 766 (e.g., image data associated with an image) and may execute instructions 768 (e.g., a computer vision application) .
  • the processor 710 may be coupled to one or more memory devices, such as a memory 732.
  • the memory 732 may store instructions, data, and other information accessible by the processor 710, such as the information 122.
  • the memory 732 may store information representing a foreground sample availability map 772 (e.g., the map 200 of FIG. 2) , a background sample availability map 774 (e.g., the map 300 of FIG. 3) , a foreground sample clearance map 776 (e.g., the map 400 of FIG. 4) , and/or a background sample clearance map 778 (e.g., the map 500 of FIG. 5) .
  • a foreground sample availability map 772 e.g., the map 200 of FIG. 2
  • a background sample availability map 774 e.g., the map 300 of FIG. 3
  • a foreground sample clearance map 776 e.g., the map 400 of FIG.
  • a background sample clearance map 778 e.g., the map 500
  • the electronic device 700 may further include a camera controller 790, which may be coupled to a camera 792 (e.g., the camera 104 of FIG. 1) .
  • the camera 792 is configured to capture an image to generate the image data 106 of FIG. 1.
  • the data 766 may include the image data 106 of FIG. 1.
  • FIG. 7 also shows a display controller 726 that is coupled to the processor 710 and to a display 728.
  • a coder/decoder (CODEC) 734 can also be coupled to the processor 710.
  • a speaker 736 and a microphone 738 can be coupled to the CODEC 734.
  • FIG. 7 also indicates that a wireless interface 740, such as a wireless controller and/or a transceiver, can be coupled to the processor 710 and to an antenna 742.
  • the processor 710, the display controller 726, the memory 732, the CODEC 734, the wireless interface 740, and the camera controller 790 are included within a system-in-package device and/or within a system-on-chip device, such as a system-on-chip device 722.
  • a system-on-chip device such as a system-on-chip device 722.
  • an input device 730 and a power supply 744 may be coupled to the system-on-chip device 722.
  • the display 728, the input device 730, the speaker 736, the microphone 738, the antenna 742, the power supply 744, and the camera 792 are external to the system-on-chip device 722.
  • each of the display 728, the input device 730, the speaker 736, the microphone 738, the antenna 742, the power supply 744, and the camera 792 can be coupled to a component of the system-on-chip device 722, such as to an interface or to a controller (e.g., to the display controller 726 or to the camera controller 790, as illustrative examples) .
  • an apparatus includes a segmenting module (e.g., the segmenting module 108) configured to identify a first region (e.g., the first region 110a) of an image, a second region (e.g., the second region 110b) of the image, and a third region (e.g., the third region 110c) of the image.
  • the apparatus further includes a raster scanner (e.g., the raster scanner 120) configured to generate information (e.g., the information 122) related to the image using raster scans of image data (e.g., the image data 106) corresponding to the image.
  • the apparatus further includes a selector (e.g., the selector 124) configured to select, based on the information, a search area (e.g., pixels of the LoS 178 and/or pixels of the LoS 179) for a line-of-sight search of the image data.
  • a selector e.g., the selector 124 configured to select, based on the information, a search area (e.g., pixels of the LoS 178 and/or pixels of the LoS 179) for a line-of-sight search of the image data.
  • an apparatus includes means for identifying (e.g., the segmenting module 108) a first region (e.g., the first region 110a) of an image, a second region (e.g., the second region 110b) of the image, and a third region (e.g., the third region 110c) of the image.
  • the apparatus further includes means for generating (e.g., the raster scanner 120) information (e.g., the information 122) related to the image using raster scans of image data (e.g., the image data 106) corresponding to the image.
  • the apparatus further includes means for selecting (e.g., the selector 124) , based on the information, a search area (e.g., pixels of the LoS 178 and/or pixels of the LoS 179) for a line-of-sight search of the image data.
  • a search area e.g., pixels of the LoS 178 and/or pixels of the LoS 179 for a line-of-sight search of the image data.
  • a computer-readable medium e.g., the memory 732 stores instructions (e.g., the instructions 768) executable by a processor (e.g., the processor 710) to perform operations during image processing of image data (e.g., the image data 106 and/or the data 766) of an image.
  • the operations include identifying a first region (e.g., the first region 110a) of the image, a second region (e.g., the second region 110b) of the image, and a third region (e.g., the third region 110c) of the image.
  • the operations further include generating information (e.g., the information 122) related to the image using raster scans of the image data.
  • the operations further include selecting, based on the information, a search area (e.g., pixels of the LoS 178 and/or pixels of the LoS 179) for a line-of-sight search of the image data.
  • a search area e.g., pixels of the LoS
  • the foregoing disclosed devices and functionalities may be designed and represented using computer files (e.g. RTL, GDSII, GERBER, etc. ) .
  • the computer files may be stored on computer-readable media. Some or all such files may be provided to fabrication handlers who fabricate devices based on such files. Resulting products include wafers that are then cut into die and packaged into integrated circuits (or “chips” ) . The chips are then employed in electronic devices, such as the electronic device 700 of FIG. 7.
  • FIGS. 1-7 illustrate certain aspects in accordance with the disclosure, the disclosure is not limited to these illustrated systems, apparatuses, and/or methods.
  • one or more functions or components described with reference to FIGS. 1-7 may be combined with one or more other features of another of FIGS. 1-7. Accordingly, no single example should be construed as limiting and examples of the disclosure may be suitably combined without departing from the teachings of the disclosure.
  • the segmenting module 108, the sampler 114, the raster scanner 120, the selector 124, and the LoS searcher 128 may be implemented using hardware, instructions stored at a non-transitory computer-readable medium, or a combination thereof.
  • the processor 710 of FIG. 7 may include one or more of the segmenting module 108, the sampler 114, the raster scanner 120, the selector 124, or the LoS searcher 128, whether such components are implemented using hardware, instructions stored at a non-transitory computer-readable medium, or a combination thereof. It is also noted that the processor 710 may perform operations of the process described with reference to Table 3 and/or operations of the method 600, such as by executing the instructions 768.
  • a software module may reside in random access memory (RAM) , flash memory, read-only memory (ROM) , programmable read-only memory (PROM) , erasable programmable read-only memory (EPROM) , electrically erasable programmable read-only memory (EEPROM) , registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM) , or any other form of non-transitory storage medium known in the art.
  • RAM random access memory
  • ROM read-only memory
  • PROM programmable read-only memory
  • EPROM erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • registers hard disk, a removable disk, a compact disc read-only memory (CD-ROM) , or any other form of non-transitory storage medium known in the art.
  • An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium.
  • the storage medium may be integral to the processor.
  • the processor and the storage medium may reside in an application-specific integrated circuit (ASIC) .
  • the ASIC may reside in a computing device or a user terminal.
  • the processor and the storage medium may reside as discrete components in a computing device or user terminal.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un procédé de traitement d'image exécuté par un dispositif qui consiste à identifier une première région d'une image, une deuxième région de l'image, et une troisième région de l'image. Le procédé consiste en outre à générer des informations relatives à l'image à l'aide de balayages de trame de données d'image correspondant à l'image. En fonction des informations, une zone de recherche est sélectionnée pour une recherche de ligne de visée des données d'image.
PCT/CN2014/093815 2014-12-15 2014-12-15 Recherche de ligne de visée à l'aide des informations de balayage de trame WO2016095081A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2014/093815 WO2016095081A1 (fr) 2014-12-15 2014-12-15 Recherche de ligne de visée à l'aide des informations de balayage de trame

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2014/093815 WO2016095081A1 (fr) 2014-12-15 2014-12-15 Recherche de ligne de visée à l'aide des informations de balayage de trame

Publications (1)

Publication Number Publication Date
WO2016095081A1 true WO2016095081A1 (fr) 2016-06-23

Family

ID=56125548

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/093815 WO2016095081A1 (fr) 2014-12-15 2014-12-15 Recherche de ligne de visée à l'aide des informations de balayage de trame

Country Status (1)

Country Link
WO (1) WO2016095081A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1310912A2 (fr) * 2001-07-26 2003-05-14 Canon Kabushiki Kaisha Procédé, appareil et système pour le traitement d'images
CN101860664A (zh) * 2010-06-13 2010-10-13 华中科技大学 一种对光照变化鲁棒的运动目标检测方法
US20110109720A1 (en) * 2009-11-11 2011-05-12 Disney Enterprises, Inc. Stereoscopic editing for video production, post-production and display adaptation
US20110274366A1 (en) * 2010-05-07 2011-11-10 Microsoft Corporation Depth map confidence filtering
CN102609723A (zh) * 2012-02-08 2012-07-25 清华大学 一种基于图像分类的全自动视频分割方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1310912A2 (fr) * 2001-07-26 2003-05-14 Canon Kabushiki Kaisha Procédé, appareil et système pour le traitement d'images
US20110109720A1 (en) * 2009-11-11 2011-05-12 Disney Enterprises, Inc. Stereoscopic editing for video production, post-production and display adaptation
US20110274366A1 (en) * 2010-05-07 2011-11-10 Microsoft Corporation Depth map confidence filtering
CN101860664A (zh) * 2010-06-13 2010-10-13 华中科技大学 一种对光照变化鲁棒的运动目标检测方法
CN102609723A (zh) * 2012-02-08 2012-07-25 清华大学 一种基于图像分类的全自动视频分割方法及装置

Similar Documents

Publication Publication Date Title
US10957093B2 (en) Scene-based foveated rendering of graphics content
CN110136082B (zh) 遮挡剔除方法、装置及计算机设备
US9208403B1 (en) Systems and methods for processing image data associated with line detection
US10438086B2 (en) Image information recognition processing method and device, and computer storage medium
US11620757B2 (en) Dense optical flow processing in a computer vision system
WO2018058595A1 (fr) Méthode et dispositif de détection de cible, et système informatique
US9147123B2 (en) System and method to perform feature detection and to determine a feature score
US10666874B2 (en) Reducing or eliminating artifacts in high dynamic range (HDR) imaging
EP3085072A1 (fr) Système et procédé pour stabiliser l'affichage d'un cadre de suivi d'objet
US8693739B2 (en) Systems and methods for performing facial detection
CN111357034A (zh) 点云生成方法、系统和计算机存储介质
US20180061070A1 (en) Image processing apparatus and method of controlling the same
WO2019201029A1 (fr) Procédé et appareil de mise à jour de boîte candidate
JP5414620B2 (ja) 画像処理装置、画像処理方法、プログラム、及び集積回路
JPWO2013179560A1 (ja) 画像処理装置および画像処理方法
WO2015031350A1 (fr) Systèmes et procédés d'utilisation de mémoire pour la détection d'objets
WO2016095081A1 (fr) Recherche de ligne de visée à l'aide des informations de balayage de trame
US10558882B2 (en) Performing distance-based feature suppression
US10062149B2 (en) Methods for blending resembling blocks and apparatuses using the same
CN117036490B (zh) 摄像机预置位偏移检测方法、装置、计算机设备与介质
US11580617B2 (en) Method of matching images to be merged and data processing device performing the same
US11113564B2 (en) Performing distance-based feature suppression
JP2011254233A (ja) 画像処理装置および方法、並びに、プログラム
JP2017072940A (ja) 画像処理システム、画像処理方法および画像処理プログラム
CN113888608A (zh) 一种目标追踪方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14908116

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14908116

Country of ref document: EP

Kind code of ref document: A1