JP2020042759A - Real time overlay arrangement in videos for augmented reality applications - Google Patents

Abstract

To provide a system and method for optimally placing context information for optimally placing an Augmented Reality (AR) application to overcome shield limitations of objects/scenes of interest through optimal placements of labels aiding better interpretations on the scene.SOLUTION: A processor implementation method is achieved by calculating an updated overlay position of a label arrangement in a video with a saliency map calculated as to each frame of an input video combined with an Euclidean distance between a current location of each frame based on an initial overlay position of labels and a previous entire position thereof. The overlay position is formulated as an objective function that minimizes visual saliency around the object of interest and minimizes time jitter, and facilitates coherence in real-time AR applications.SELECTED DRAWING: Figure 2

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the complete specification of Indian Patent Application No. 201210333541, filed in India on September 6, 2018, entitled "Real-Time Overlay Arrangement in Video for Augmented Reality Applications." Insist.

  The present disclosure relates generally to video analysis, and more particularly to systems and methods for real-time overlay placement in video for augmented reality applications.

  Augmented reality (AR) with virtual reality (VR) is considered the fourth wave of technology after personal computers (PCs), the Internet, and mobile. AR extends real-world scenes by superimposing virtual information to enable better situational awareness and enhance human perception and perception. This context information may take the form of, but not limited to, text, 3D objects, GPS coordinates, and audio. The placement of such contextual information is an important contribution to scene understanding, a key issue in artificial intelligence. Spatial placement of labels is a challenging task due to the constraints that the labels are (i) not obstructing the object / scene of interest and (ii) are optimally placed for better interpretation of the scene. Advanced state-of-the-art techniques for optimally positioning text labels work only on images and are inefficient for real-time implementation on devices (eg, mobile communication devices such as smartphones, tablets, etc.). There are many.

  Embodiments of the present disclosure present a technical improvement as a solution to one or more of the above-mentioned technical problems identified by the inventors in conventional systems. For example, in one aspect, a processor-implemented method for real-time overlay placement in video for augmented reality applications is provided. The method includes the steps of: (i) input video including a plurality of frames and an object in the plurality of frames; and (ii) a label in which an initial overlay position is pre-calculated for an arrangement of the input video on a center frame in real time. Receiving, calculating a saliency map for each of the plurality of frames in real time to obtain a plurality of saliency maps, and obtaining each of the plurality of Euclidean distances, for each of the plurality of frames, Real-time calculation of the Euclidean distance between the current overlay position and the previous overlay position based on the label's initial overlay position and placement in the input video based on multiple saliency maps and multiple Euclidean distances The updated overlay position of the label for And a be.

  In one embodiment, the updated overlay position of the label can be calculated by combining multiple saliency maps and multiple Euclidean distances.

  In one embodiment, the Euclidean distance of each of the plurality of frames is calculated to control the time jitter at the position of the label located in the input video in real time. In one embodiment, the method can further include shifting the label from the initial overlay position to the updated overlay position to minimize obstruction of the object.

  In one embodiment, a plurality of pixels within a predetermined threshold range and corresponding to the Euclidean distance between the current overlay position and the previous overlay position shift the label from the initial overlay position to the updated overlay position. Selected for.

  For example, in one aspect, a system is provided for real-time overlay placement in video for augmented reality applications. The system includes a memory for storing instructions, one or more communication interfaces, and one or more hardware processors coupled to the memory via the one or more communication interfaces. The plurality of hardware processors are pre-computed by the instructions with an initial overlay position for (i) an input video including a plurality of frames and objects in the plurality of frames, and (ii) an arrangement of the input video on a central frame. Receiving real-time labels, calculating a saliency map for each of a plurality of frames in real time to obtain a plurality of saliency maps, and obtaining a plurality of Euclidean distances. Current frame based on the label's initial overlay position for each of the frames -Real-time calculation of the Euclidean distance between the overlay position and the previous overlay position, and an updated overlay position of the label for placement in the input video based on multiple saliency maps and multiple Euclidean distances Is calculated in real time.

  In one embodiment, the updated overlay position of the label is calculated by combining multiple saliency maps and multiple Euclidean distances. In one embodiment, the Euclidean distance of each of the plurality of frames is calculated to control the time jitter at the position of the label located in the input video in real time.

  In one embodiment, the one or more hardware processors are further configured to shift the label from the initial overlay position to the updated overlay position to minimize obstruction of object viewing. You. In one embodiment, a plurality of pixels within a predetermined threshold range and corresponding to the Euclidean distance between the current overlay position and the previous overlay position shift the label from the initial overlay position to the updated overlay position. Selected for.

  In yet another aspect, one or more instructions that include one or more instructions that, when executed by one or more hardware processors, provide a method for real-time overlay placement in video for augmented reality applications. A non-transitory machine-readable information storage medium is provided. The instructions receive in real time (i) an input video including a plurality of frames and an object in the plurality of frames, and (ii) a label whose initial overlay position is pre-computed for placement on the center frame of the input video. Calculating a saliency map for each of the plurality of frames in real time to obtain a plurality of saliency maps; and labeling each of the plurality of frames to obtain a plurality of Euclidean distances. Calculating in real time the Euclidean distance between the current overlay position and the previous overlay position based on the initial overlay position of the input and placing it in the input video based on the saliency maps and the Euclidean distances The updated overlay position of the label in real time Bring and that.

  In one embodiment, the updated overlay position of the label can be calculated by combining multiple saliency maps and multiple Euclidean distances.

  In one embodiment, the Euclidean distance of each of the plurality of frames is calculated to control the time jitter at the position of the label located in the input video in real time. In one embodiment, the instructions, when executed by one or more hardware processors, move the label from the initial overlay location to the updated overlay location to minimize obstruction of object viewing. Bringing the shift further.

  In one embodiment, a plurality of pixels within a predetermined threshold range and corresponding to the Euclidean distance between the current overlay position and the previous overlay position shift the label from the initial overlay position to the updated overlay position. Selected for.

  It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as claimed.

  The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles.

FIG. 1 is an exemplary block diagram of a system for real-time overlay placement in video for augmented reality applications, according to one embodiment of the present disclosure. 4 is an exemplary flowchart of a method for real-time overlay placement in video for augmented reality (AR) applications using the system of FIG. 1 according to one embodiment of the present disclosure. FIG. 4 is a block diagram for real-time overlay placement in input video by calculating saliency maps and Euclidean distances, according to one embodiment of the present disclosure. FIG. 6 is a graph illustrating the variation of average label occlusion versus saliency (LOS) score with changes in λ and Σ as a contour plot, according to one embodiment of the present disclosure.

  Exemplary embodiments are described with reference to the accompanying drawings. In the figures, the left-most digit (s) of a reference number identifies the figure in which the reference number first appears. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or like parts. Although examples and features of the disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. The following detailed description is to be considered only by way of example, the true scope and spirit of which is intended to be indicated by the appended claims.

  As mentioned above, augmented reality (AR) with virtual reality (VR) is considered the fourth wave of technology after PC, Internet and mobile. Overlaying virtual information on real-world scenes is considered to be very important to enable better situational awareness and enhance human perception and perception. The placement of such contextual information is an important contribution to scene understanding, a key issue in artificial intelligence.

  Some of the uses related to optimal placement of text labels are: (I) Optimal placement of advertisements in indoor / outdoor scenes and videos is a very important advertising strategy to get the viewer's visual attention. (Ii) Labels identifying the names of nearby monuments and buildings help the traveler better recognize the situation. (Iii) Various conventional applications enable real-time translation on various operating systems (e.g., Android devices) by using the camera. Note that AR applications on mobile phones help to perform tasks faster, more accurately, efficiently and with lower cognitive load. Another example where optimal overlay placement may be useful is the situation where a soldier uses a head mounted device (HMD). The overlay of the crew's GPS coordinates in the battlefield map displayed on the HMD should not obstruct the real view of the scene when important. In addition, the optimal placement of subtitles in the video helps to avoid deviating viewpoints. Smart label placement helps to make videos interesting by using comic style overlay placement.

  The placement of these 2D text labels is difficult when the contextual information is overlaid in such a way that the overlay does not obstruct the object / scene of interest, thereby arranging for a better interpretation. Few studies have found that label placement in AR applications is not simple when placement needs to work in real time. For the simple task of placing labels on still images, the number of possible label locations increases exponentially with the number of items to be labeled. Other issues include a lack of understanding of cognitive and perceptual issues regarding label placement in AR applications.

  Having said all of the above, overlay placement around the object / scene of interest has received little attention in the visual image area compared to object detection and segmentation. Recently, as the demand for AR applications that overlay text labels in real time has increased, label placement has received a great deal of attention. Related work has been done on geometry-based and image-based layouts for rendering labels, aesthetic rules, and optimal text label placement based on adaptive overlay.

  In geometry-based layout techniques, point feature label placement has proven to be an NP-hard problem, and annealing and gradient descent have been proposed as solutions. Image aesthetic-based (or image-based layout) approaches have been developed to consider the visual aesthetics of a computer interface as a strong determinant of user satisfaction. They make use of general design principles, such as spatial layout rules, symmetry, balance between elements, and harmonization with color design and photobook generation use cases. However, the above approach operates on images and is not suitable for real-time camera streams (or real-time video streams).

  Several other studies have focused on image-driven view management for AR browsers for placement of labels on video streams using a combination of saliency maps and edge maps. In such studies, significant limitations may be encountered when applying this approach to video streams on mobile devices, some of which include, first of all, that such techniques may require a small amount of camera movement. It has been observed that at some point it applies significantly. For large movements, those techniques use a static layout for the labels. For AR-based applications, this method is clearly not possible. Second, performing the visual saliency algorithm involves computationally expensive matrix operations. This problem is particularly pronounced on mobile devices where computing resources and memory are limited. In addition, these and other previously known text overlay techniques, such as those described above, are computationally intensive, mostly operate on images on desktop computers, lack real-time performance, and are not compatible with video overlays. Not suitable. In addition, overlays have their own challenges due to occlusion, poor lighting scenarios, and scene changes in the live view.

  Embodiments of the present disclosure provide systems and methods for strategic placement of context labels for AR applications. The systems and methods of the present disclosure provide a label placement technique that works in real time even on low-end Android devices such as smartphones and tablets. In the present disclosure, label placement is formulated as an objective function parameterized by image saliency and time jitter. The present disclosure implements a label occlusion versus saliency (LOS) score calculation for saliency to measure the effectiveness of overlay placement.

  Referring now to the drawings, in which like reference numerals refer to corresponding features throughout the drawings, and more particularly to FIGS. 1-4, preferred embodiments are illustrated and these embodiments are described below. It is described in the context of an exemplary system and / or method.

  FIG. 1 illustrates an exemplary block diagram of a system 100 for real-time overlay placement in video for augmented reality applications, according to one embodiment of the present disclosure. System 100 is also referred to as an “overlay placement system” and is used interchangeably hereinafter. In one embodiment, system 100 is operable with one or more processors 104, communication interface device (s) or input / output (I / O) interface (s) 106, and one or more processors 104. And one or more data storage devices or memories 102 coupled to the One or more processors 104 may be one or more software processing modules and / or hardware processors. In one embodiment, a hardware processor manipulates signals based on one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or operating instructions. It can be implemented as any device. Among other functions, the processor (s) is configured to retrieve and execute computer readable instructions stored in memory. In one embodiment, device 100 may be implemented in various computing systems such as laptop computers, notebooks, handheld devices, workstations, mainframe computers, servers, network clouds, and the like.

  The I / O interface device (s) 106 may include various software and hardware interfaces, such as, for example, a web interface, a graphical user interface, and the like, for example, a wired network such as a LAN, cable, WLAN, cellular, satellite, etc. Communication in a wide variety of network N / W and protocol types, including wireless networks of In one embodiment, the I / O interface device (s) may include one or more ports for connecting several devices to each other or to another server.

  Memory 102 may be volatile memory such as, for example, static random access memory (SRAM) and dynamic random access memory (DRAM), and / or read only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk, and It can include any computer readable medium known in the art, including non-volatile memory such as magnetic tape. In one embodiment, database 108 may be stored in memory 102, which may include, but is not limited to, information input videos, frames, objects, labels, initial overlay positions of labels, label width and height, salient. Gender map output, Euclidean distance output (s), and updated overlay locations for placement in the video. More specifically, it includes information about the input video, including pixel information, current and previous overlay positions for each frame, time jitter, predetermined threshold ranges, and the like. In one embodiment, memory 102 is one or more techniques (one or more) for performing the methodology described herein when executed by one or more hardware processors 104 (e.g., , Saliency map calculation technique (s), Euclidean distance calculation technique (s)). Memory 102 may further include information regarding the input (s) / output (s) of each step performed by the systems and methods of the present disclosure.

FIG. 2 is a flowchart of a method for real-time overlay placement in video for an augmented reality (AR) application using the system 100 of FIG. 1 according to an embodiment of the present disclosure with reference to FIG. 1. Is shown. In one embodiment, the system (s) 100 is operatively coupled to one or more hardware processors 104 and stores instructions for performing the method steps by one or more processors 104. One or more data storage devices or memories 102 configured to The steps of the method of the present disclosure will now be described with reference to the components of the system 100 as shown in FIG. 1 and a flow chart as shown in FIG. Prior to receiving the input video in real time (also referred to as "real-time" and sometimes used interchangeably hereinafter), the system 100 and associated methods require the user (s) to )), I.e., k, λ, Σ, O _h , and O _w . Where:
1) k is the number of frames whose processing is to be skipped. The techniques / methods of the present disclosure are performed every k frames. If k = 1, the method of the present disclosure is performed on every frame. Similarly, for k = 2, the method is performed on every other frame.
2) λ controls the temporal coherence of the subsequent overlay. A small value of λ means that the overlay is likely to be located in areas where the overlay is less pronounced, but it will also be subject to more jitter. Higher values of λ reduce jitter, but also limit overlay movement.
3) Σ is a search space sampling parameter. This uniformly samples the pixels in the two-dimensional image space. For example, consider u _w and u _h to be the width and height of the frame, respectively. These are the dimensions of the search space in the context of the present invention. Then, u _h / _ｈ and u _w / Σ pixels are skipped in each image dimension.
4) O _h and O _w are the overlay height and overlay width, respectively.

It may not be feasible for the techniques or methods of this disclosure to search all pixel values to calculate the best overlay position. The saliency map has discrete values, so it may not be possible to use optimization techniques such as stochastic gradient descent. A linear search through all pixels is prohibitively expensive. In the present disclosure, a uniform sampling approach is taken. Some other intermediate variables calculated by the method and system 100 of the present disclosure are as follows.
a) X _P ; Y _P is the optimal position of the overlay in the previous iteration. It is initialized to the center of the frame.
b) X; Y is the optimal position of the overlay calculated in the current iteration.
c) SM uses conventional computational technique (s) (e.g., Radhakrishna Achanta, also referred to as Achanta et al., Sheila Hemami, Francisco Estrada, and Sabine Sustain der kontagion responsibilities of "Frequency-tuned responsibilities"). 2009. cvpr 2009. IEEE conference on.IEEE, 2009, pp. 1597-1604), or conventional visual saliency techniques, which may be used interchangeably herein. It is a saliency map calculated.
d) P is a set of pixels sampled from the search space.
e) F _w and F _h are the width and height of the video frame, respectively.

  The above description is better understood by the following steps set forth in FIG. In one embodiment of the present disclosure, in step 202, one or more hardware processors 104 include: (i) an input video including a plurality of frames and an object in the plurality of frames; Receive in real time a label with an initial overlay position pre-computed for placement on the center frame. In one embodiment, the label includes a label height and a label width. In one embodiment of the present disclosure, the input video is shown in FIG. A label having an initial overlay position (eg, the label will be on or located at the center frame of the input video) is also received as input (not shown in FIGS. 2 and 3). . Upon receiving the input video and label, at step 204, one or more hardware processors calculate a saliency map for each of the plurality of frames in real time to obtain a plurality of saliency maps. An example saliency map is shown in FIG. In the present disclosure, system 100 calculates a saliency map for each frame present in the input video. In other words, there will be one saliency map for each corresponding frame of the input video. Therefore, the saliency map calculation is performed iteratively until the last frame of the input video to obtain a plurality of saliency maps.

  At step 206, one or more hardware processors 104 determine in real time the Euclidean distance between the current overlay position and the previous overlay position based on the label's initial overlay position to obtain a plurality of Euclidean distances. calculate. The Euclidean distance calculation is performed iteratively until the last frame of the input video to obtain multiple Euclidean distances. In other words, the Euclidean distance is calculated for each of the plurality of frames. In other words, as in the case of the saliency map calculation, there will be one Euclidean distance for each corresponding frame of the input video. An exemplary Euclidean distance calculation is shown in FIG. In the present disclosure, the Euclidean distance is calculated for each frame to control the time jitter at the location of the label that will be placed in the input video. Control of the time jitter occurs in real time as the input video is received and processed in real time.

  Once the saliency maps and the Euclidean distances have been calculated, at step 208, one or more hardware processors 104 place in the input video based on the saliency maps and the Euclidean distances. The updated overlay position of the label to calculate in real time. In other words, the updated overlay position of the label is calculated by combining multiple saliency maps and multiple Euclidean distances, as shown in FIG. Steps 204 and 206 are performed sequentially, but (i) calculating a saliency map for each of the plurality of frames; and (ii) determining a current overlay position and a previous overlay position for each of the plurality of frames. And calculating the Euclidean distance between can be performed simultaneously. This may make the calculation in a shorter time more reliable, which may result in better or optimal resource utilization. Further, once the updated overlay position has been calculated, system 100 (or one or more hardware processors 104) minimizes (or reduces) obstruction of the object at step 210. ) To shift the label from the initial overlay position to the updated overlay position. Alternatively, this observation also ensures that the observation is unobstructed when the label shifts from the initial overlay position to the updated overlay position. In the present disclosure, a plurality of pixels corresponding to a Euclidean distance that is within a predetermined threshold range between a current overlay position and a previous overlay position shift a label from an initial overlay position to an updated overlay position. Is selected. In other words, one or more of the Euclidean distances between the current overlay position and the previous overlay position are within a predetermined threshold range (also referred to as a “predetermined threshold” and may be used interchangeably below). Pixels are selected to shift the label from its initial overlay position to an updated overlay position calculated by the system 100 in real time. The updated overlay position includes information about the label with label height and label width (eg, where the width and height can be the same as the initial width and height associated with the initial overlay position, or May vary depending on the selection of multiple pixels). An exemplary overlay frame is shown in FIG. More specifically, FIG. 3 is a block diagram for real-time overlay placement in input video by calculating saliency maps and Euclidean distances, according to one embodiment of the present disclosure, with reference to FIGS. 1-2. Is shown.

  In short, steps 202 through 208 are described as follows for a better understanding.

The method of the present disclosure is performed, for example, every k frames. For a given frame, a visual saliency map (also referred to as a saliency map, sometimes used interchangeably) using pseudo-code (eg, SaliencyMapComputation) is calculated. . Next, the system 100 iterates through the pixel values provided in the search space (see, for example, Σ search space sampling parameters), and the saliency value given by the map in a virtual box of size O _h , O _w Sum up. In the present disclosure, the pixel value with the lowest sum is selected as the ideal candidate indicating the lowest saliency. If the Euclidean distance d between the previous position and the current position, scaled by λ (referred to as a predetermined threshold range or predetermined threshold), is as small as possible, the overlay is shifted. To combine the constraints imposed by both low saliency and time jitter, the present disclosure formulates an optimization problem as follows.

The following is exemplary pseudo-code of the techniques / methods of the present disclosure.
1. _{(X P; Y P) =} ( Frame Width / 2, the frame height / 2)
2. 2. Every k frames SM = saliency map calculation (frame)
4. for (x, y) ∈P
5. L = {(a, b) | x ≦ a ≦ x + O _w , y ≦ b ≦ y + O _h }
6. s _{x, y} = Σ _{(a, b) ∈L} SM (a, b)
7. d _{x, y} = λ × distance ((X, Y), (X _P , Y _P ))
8. s _min = min (s _{x, y} + d _{x, y} )
9. (X, Y): = arg_min (s _{x, y} )
10. (X _P , Y _P ): = (X, Y) // Use linear interpolation for entire transition

  In the above pseudo code, in order to execute the line (or command or program code) “SM = saliency map calculation (frame)”, it is possible to refer to a conventional saliency map calculation technique (for example, Radakrishna). Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk "Frequency-tuned salient region detection" (Computer vision and pattern recognition, 2009.cvpr 2009.IEEE conference on.IEEE, 2009, pp.1597-1604 .'- also referred as Achanta et al. hich can be found at-https: //infoscience.epfl.ch/record/135217/files/1708.pdf) see). More specifically, in one embodiment, for a better understanding of saliency map calculations, section 3 of Achanta et al., Supra, including equations (1), (2), (3) and (4), .2 can be referred to.

Experiments and Results Experiments included subjects (eg, 25 researchers in the 25-34 age group to test methods / pseudocodes) to observe the object under inspection by a 3D printer through a tablet. , 10 women and 15 men). A series of subjective and objective metrics were acquired to evaluate (a) user experience and (b) overlay placement. Labels with dimensions 50x50 were used in all experiments. It can be customized according to the needs of the user. The experiments were performed on Nexus® 6 Android phone and Nexus® 9 tablet. Users were required to rate the following parameters on a scale of 1-5. Thereafter, an average opinion score was obtained. The metrics used were (i) the location of the overlay, (ii) low jitter within the overlay, (iii) the color of the overlay box and text, and (iv) the overlay responsiveness.

  The present disclosure used DIEM datasets to evaluate the methods of the present disclosure (eg, Parag K Mital, Tim J. Smith, Robin L Hill, and John M Henderson, "Clustering of gadgeticing dancing dynamics"). motion "(Cognitive Computation, vol.3, no.1, pp.5-24,2011.) -http: //pkmital.com/home/wp-content/uploads/2010/03/Mital_Clustering_of_Gaze_During_Dynamic_Scene_Viewing_is_Predicted.p See f). To perform the experiments according to the present disclosure, a video with a resolution of 1280 × 720 was taken from the dataset. This dataset consisted of various genres of advertisements, trailers, and various videos of television series. Also, using eye movements, this dataset provided detailed eye fixation saliency annotations.

  During the experiment, the values of the parameters λ and Σ were empirically found on a DIEM dataset from a grid search (known in the art), and the average label occlusion versus saliency (LOS) score of the overlay over the video (LOS) (Defined and discussed below). FIG. 4 shows, with reference to FIGS. 1-3, a graph illustrating the variation of average label occlusion versus saliency (LOS) score with changes in λ and Σ as a contour plot, according to one embodiment of the present disclosure. . More specifically, FIG. 4 shows a contour plot of the average LOS score for λ and Σ. During the experiment, the LOS score was independent of λ, the optimal combination of Σ and λ was (5, 0.021), and it was observed that Σ is preferably small (indicated by 402; (See line representation with symbols such as inverted Y between contours in FIG. 4).

Results Subjective Metrics The following exemplary table (Table 1) shows the subjective metric scores.

From Table 1 above, it is inferred by the present disclosure that the position of the overlay was rated very high at 4.5, which is crucial to prevent the overlay from covering a significant area in the scene. A real-time implementation of the above pseudo-code of the method of the present disclosure performed at approximately 20 frames per second (fps) has resulted in an overlay responsive score of 4.7 in some cases. Simple color design-a white box with black font and vice versa was chosen, the color of the box was set to α = 0.5 for transparency. The color of the overlay box relied on a simple adaptive threshold applied to the pixel intensity (luminance channel Y) given by the exemplary equation (2) below.

  The data drive threshold Thresh is the average of the difference between the maximum and minimum luminance values for a given scene. If this value is greater than or equal to Threshold, the overlay box uses a black background and vice versa (the entire setting of how text labels are overlaid is noted, but this has been demonstrated through experimentation). Please note that). The sample overlay under consideration during the experiment showed only contextual information about the entire scene. The demonstration also showed that the overlay worked in real time, while at the same time having low jitter.

Objective Metrics The effectiveness of overlay placement performed by the methods / pseudocode of the present disclosure was compared. The criterion for this comparison was based on the average LOS score occluded by labels with video saliency ground truth. A lower score indicates an effective overlay arrangement with less occlusion. The label occlusion versus saliency (LOS) score S is defined and expressed as the following equation.

Where L is the set of pixels (x, y) occluded by the overlay and G is the ground truth saliency map. The above pseudo-code for the method of the present disclosure was found to have an average LOS score of 0.042 and take 0.021 seconds to calculate the overlay position.

  Embodiments of the present disclosure provide systems and methods for real-time overlay (context information) placement in video for AR applications. Based on the above experiments and results, it is observed that the present disclosure overcomes the limitations of object / object scene occlusion by optimally placing labels that aid in better interpretation of the scene. The placement of the overlay is formulated as an objective function that minimizes (i) visual saliency around the object and (ii) time jitter, and is used for real-time AR applications (especially (low-end or high-end) smartphones, tablets ( Yes), implemented in AR-based browsers, etc.). Examples of AR applications can include, but are not limited to, navigation maps, virtual environment experiences that can be visualized in gaming applications, and the like. Other examples of AR-based applications include, but are not limited to, museum exploration work, industrial inspection and repair work, advertising and media, and live situation awareness in the tourism industry.

  This specification describes the subject matter herein to enable one of ordinary skill in the art to make and use the embodiments. The scope of the subject embodiments is defined by the claims, and may include other modifications that occur to those skilled in the art. Such other changes are within the scope of the claims, if they have similar elements that do not differ from the claim language, or if they include equivalent elements that have slight differences from the claim language. It is intended to be

  It is to be understood that the scope of protection extends to such programs and, in addition, to computer readable means having messages therein. Such computer readable storage means include program code means for performing one or more steps of the method when the program is executed on a server or mobile device or any suitable programmable device. . A hardware device can be any type of programmable device, including, for example, any type of computer, such as a server or personal computer, or any combination thereof. The device can also be a hardware means, such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a combination of hardware and software means, such as, for example, ASICs and FPGAs; Alternatively, it may be at least one memory with at least one microprocessor and software module located therein. Thus, the means may include both hardware and software means. The method embodiments described herein can be implemented in hardware and software. The device may also include software means. Alternatively, embodiments may be implemented on different hardware devices, for example, using multiple CPUs.

  Embodiments herein may include hardware and software elements. Embodiments implemented in software include, but are not limited to, firmware, resident software, microcode, and the like. The functions performed by the various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this specification, computer usable or computer readable media includes, stores, communicates, propagates, including programs for use by or in connection with an instruction execution system, apparatus, or device. Or any device that can be transported.

  The illustrated steps are set forth to illustrate the exemplary embodiments shown, and it is expected that ongoing technology development will change the way certain functions are performed. These examples are provided herein for illustrative purposes and are not limiting. Further, boundaries of functional building blocks are arbitrarily defined herein for the convenience of description. Alternative boundaries can be defined as long as the particular functions and their relationships are properly performed. Alternatives, including equivalents, extensions, variations, departures, and the like, of those described herein will be apparent to those skilled in the art based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments. Also, the words “comprising”, “having”, “containing”, and “including”, and other similar forms, are equivalent in meaning and are preceded by one of the following: Or where the items are not intended to be an exhaustive list of one or more of such items, or are limited to only one or more of the listed items. It is intended to be unlimited in the sense that it is not intended. As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Please note also.

  Further, one or more computer-readable storage media can be utilized in implementing embodiments consistent with the present disclosure. Computer readable storage media refers to any type of physical memory in which information or data readable by a processor can be stored. Accordingly, the computer-readable storage medium includes instructions for execution by one or more processors, including instructions for causing processor (s) to perform steps or steps consistent with embodiments described herein. Can be stored. The term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transients, ie, non-transitory. Examples include random access memory (RAM), read only memory (ROM), volatile memory, non-volatile memory, hard drive, CD-ROM, DVD, flash drive, disk, and any other known physical storage Media.

  It is intended that the disclosure and examples be considered as exemplary only, with a true scope and spirit of the disclosed embodiments being indicated by the following claims.

REFERENCE SIGNS LIST 100 system, device 102 memory 104 hardware processor 106 input / output (I / O) interface 108 database

Claims (15)

A processor implementation method, comprising:
Receive in real time: (i) an input video that includes a plurality of frames and objects within the plurality of frames; and (ii) a label whose initial overlay position is pre-computed for an arrangement of the input video on a central frame. (202)
Calculating in real time a saliency map for each of the plurality of frames to obtain a plurality of saliency maps;
Calculating, in real time, a Euclidean distance between a current overlay position and a previous overlay position based on the initial overlay position of the label for each of the plurality of frames to obtain a plurality of Euclidean distances ( 206),
Calculating, in real time, an updated overlay position of the label for placement in the input video based on the plurality of saliency maps and the plurality of Euclidean distances;
A processor implementation method, comprising:   The processor-implemented method of claim 1, wherein the updated overlay position of the label is calculated by combining the saliency maps and the Euclidean distances.   The processor-implemented method of claim 1, wherein the Euclidean distance of each of the plurality of frames is calculated to control a time jitter at a position of the label located in the input video in real time.   The processor implementation of claim 1, further comprising: shifting (210) the label from the initial overlay location to the updated overlay location to minimize obstruction of viewing of the object. Method.   A plurality of pixels corresponding to a Euclidean distance within the predetermined threshold range between the current overlay position and the previous overlay position for shifting a label from the initial overlay position to the updated overlay position. The method of claim 1, wherein the method is selected. A memory (102) for storing instructions,
One or more communication interfaces (106);
One or more hardware processors (104) coupled to the memory (102) via the one or more communication interfaces (106), wherein the system (100) comprises:
The one or more hardware processors (104), according to the instructions,
Receive in real time: (i) an input video that includes a plurality of frames and objects within the plurality of frames; and (ii) a label whose initial overlay position is pre-computed for an arrangement of the input video on a central frame. That
Calculating a saliency map for each of the plurality of frames in real time to obtain a plurality of saliency maps;
Calculating, in real time, a Euclidean distance between a current overlay position and a previous overlay position based on the initial overlay position of the label, for each of the plurality of frames, to obtain a plurality of Euclidean distances. ,
Calculating, in real time, an updated overlay position of the label for placement in the input video based on the plurality of saliency maps and the plurality of Euclidean distances;
The system (100), wherein the system (100) is configured to:   7. The system of claim 6, wherein the updated overlay position of the label is calculated by combining the plurality of saliency maps and the plurality of Euclidean distances.   7. The system of claim 6, wherein the Euclidean distance of each of the plurality of frames is calculated to control a time jitter at a position of the label located in the input video in real time.   The one or more hardware processors are further configured to shift the label from the initial overlay position to the updated overlay position to minimize obstruction of viewing of the object. The system of claim 6, wherein   A plurality of pixels corresponding to a Euclidean distance within the predetermined threshold range between the current overlay position and the previous overlay position for shifting a label from the initial overlay position to the updated overlay position. 7. The system of claim 6, wherein the system is selected. When executed by one or more hardware processors,
Receive in real time: (i) an input video that includes a plurality of frames and objects within the plurality of frames; and (ii) a label whose initial overlay position is pre-computed for an arrangement of the input video on a central frame. That
Calculating a saliency map for each of the plurality of frames in real time to obtain a plurality of saliency maps;
Calculating, in real time, a Euclidean distance between a current overlay position and a previous overlay position based on the initial overlay position of the label, for each of the plurality of frames, to obtain a plurality of Euclidean distances. ,
Calculating, in real time, an updated overlay position of the label for placement in the input video based on the plurality of saliency maps and the plurality of Euclidean distances;
One or more non-transitory machine-readable information storage media comprising one or more instructions that provide for:   The one or more non-transitory machine-readable information storage of claim 11, wherein the updated overlay position of the label is calculated by combining the plurality of saliency maps and the plurality of Euclidean distances. Medium.   The one or more non-Euclidean distances of claim 11, wherein the Euclidean distance of each of the plurality of frames is calculated to control, in real time, a time jitter at a position of the label located in the input video. Temporary machine-readable information storage medium.   The instructions, when executed by the one or more hardware processors, move the label from the initial overlay location to the updated overlay location to minimize obstruction of viewing of the object. 12. The one or more non-transitory machine-readable information storage media of claim 11, further comprising:   A plurality of pixels corresponding to a Euclidean distance within the predetermined threshold range between the current overlay position and the previous overlay position for shifting a label from the initial overlay position to the updated overlay position. 12. The one or more non-transitory machine-readable information storage media of claim 11, wherein the non-transitory machine-readable information storage media is selected.