WO2019182907A1

WO2019182907A1 - Design, platform, and methods for personalized human interactions through digital communication devices

Info

Publication number: WO2019182907A1
Application number: PCT/US2019/022552
Authority: WO
Inventors: Yanir NULMAN
Original assignee: Nulman Yanir
Priority date: 2018-03-21
Filing date: 2019-03-15
Publication date: 2019-09-26

Abstract

A platform residing in the cloud, a physical server, desk-top computer, or digital mobile devices which recognizes images of objects and humans and converts them to a digital end points for the purpose of interactions between users of applications on digital communication devices such as smart phones, laptop, tablets, smart glasses, and other interactive devices.

Description

DESIGN, PLATFORM, AND METHODS FOR PERSONALIZED HUMAN INTERACTIONS THROUGH DIGITAL COMMUNICATION DEVICES

CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims benefit of U.S. Provisional Application Serial No. 62/62/646,218, filed March 21 , 2018, entitled “DESIGN, PLATFORM, AND METHODS FOR PERSONALIZED HUMAN INTERACTIONS THROUGH DIGITAL COMMUNICATION DEVICES,” which is incorporated herein by reference in its entirety.

BACKGROUND

Field

[0002] The present disclosure relates to the design of, a platform for, and methods for personalized real time direct view of human interactions through digital communication devices such as smartphones, tablets, personal computers, etc.

Description of the Related Art

[0003] The advent of digital wearable and portable Digital Communication Devices has resulted in humans using the applications developed for the use of these devices for communication between peers, as well as interaction with the ambient environment, in an artificial and indirect way. As a consequence the user looks into the device, and not at their peers and their ambient environment, and de- facto becomes disconnected from their surroundings. Interactions employing digital wearable and portable Digital Communication Devices (DCD’s) also lack privacy and security, resulting in messages being delivered to the wrong recipient, or receiving messages send by unknown entities. The graphical user interface of DCDs is used for viewing images, and visual content generated by or through the application being used, for interacting with peers, friends, professionals, relatives, businesses, and the environment in an indirect and non-personal way. In other words, an individual interacting with their environment is looking at a screen, and does not see a counterpart peer with whom they are communicating unless a video link is established. This video link lacks the needed interactivity for sharing other elements of communication between peers, is impersonal, and is not always available as communication bandwidth limits the user experience.

SUMMARY

[0004] Provided herein is a platform with applications based on objects within the viewable area of a Digital Communications Device’s camera. The platform recognizes the objects, and converts them into digital end points for use by the applications’ digital interactions, which are integrated to the platform by means of the application’s programing interfaces, which include as examples sending messages, electronic monetary transactions, sending documents, etc. The applications’ graphical user interface design, and use methods, position the user for real time personalized and direct digital interactions with the physical objects in front of them or in there, or their virtual reality devices, filed of view. The environment comprises, but is not limited to peers, friends, colleagues, family, buildings, billboards, signage, restaurants, store fronts, teachers, nature, logos, internet connected devices, digital devices, animals, automobiles, bicycles, vending machines, businesses, etc.

[0005] A platform residing in the cloud, a physical server, desk-top computer, or digital mobile devices is provided herein which recognizes images of objects and humans and converts them to a digital end points for the purpose of interactions between users of applications on digital communication devices such as smart phones, laptop, tablets, smart glasses, and other interactive devices.

BREIF DESCRIPTION OF THE FIGURES

[0001] So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments.

[0002] Figure 1 is a schematic front view of a screen of a user device.

[0003] Figure 2a is a schematic front views of the screen of the user device of Figure 1 , further including an image generated by a camera of the device onto the screen.

[0004] Figure 2b is a schematic front views of the screen of the user device of Figure 1 , further including an image generated by a camera of the device onto the screen, wherein the user device generates and displays a frame around the image

[0005] Figure 2c is a schematic front views of the screen of the user device of Figure 1 , further including an image generated by a camera of the device onto the screen, wherein the user device generates and displays a frame around the image, schematically showing a user touching the image within the frame on the screen.

[0006] Figure 2a is a schematic front views of the screen of the user device of Figure 1 , further including an image of a person generated by a camera of the device onto the screen.

[0007] Figure 2b is a schematic front views of the screen of the user device of Figure 1 , further including an image of a person generated by a camera of the device onto the screen, wherein the user device generates and displays a frame around the image

[0008] Figure 2c is a schematic front views of the screen of the user device of Figure 1 , further including an image of a person generated by a camera of the device onto the screen, wherein the user device generates and displays a frame around the image, schematically showing a user touching the image within the frame on the screen.

[0009] Figure 4 is a flow diagram of a methodology for determining whether to download an available application.

[0010] Figure 5 is a flow diagram of a methodology for determining whether to locate a downloaded application in a rewritable memory application.

[0011] Figure 6 is a flow diagram of a methodology for overlaying an image of a portion of a person over 100 an image of that person, where a portion of the person in the image is blocked.

[0012] Figure 7 is a flow diagram of a methodology for determining where, in a displayed image, to render a graphical image. [0013] Figure 8 is a flow diagram of a methodology for setting an image in a 3D mesh.

[0014] Figure 9 is a schematic view of a processing circuit of a smart device.

[0015] Figure 10 is a schematic representation of a smart device.

DETAILED DESCRIPTION

[0006] The present disclosure includes a platform, and applications useable therewith, with a graphical user interface, combined with methods of use that provide for a direct, secure, personalized and dynamic interaction between the user and their peers, friends, relatives, businesses, and the environment in general. The platform is the computational engine and data provider for the applications, and also functions to facilitate application use by end users. This platform can exist as an operating system or a software package that is stored in a device memory. Furthermore, it can communicate with a remote server for data retrieval and computation if necessary. The Personalized Interactive Application (the “PIA”) enables the user to see, i.e., visual through a screen of a virtual reality device, i.e., the graphical user interface being viewed, a body or an object with whom he or she is interacting because the graphical user interface is populated by an image of the body or object that is in the DCD camera view, or in the field of view of a digital camera connected to the DCD. Simultaneously the DCD graphical user interface displays interaction functions, or a menu of functions including interaction functions accessible through an icon that occupies between 0.5% to 70%, preferably between 1 and 40%, and more preferable between 1 to 10% of the viewing area of the graphical user interface. In this way, the user interacts facing the body or object with whom the interaction is taking place, instead of the current practice in which the user is facing downwardly, or looking at the graphical user interface, while ignoring the environment surrounding the user and the body or object with which the interaction is taking place. Thus, the platform, and applications useable therewith hereof enable a personal interaction in the way that existed before the advent of DCDs. [0007] With reference to Figure 1 , the graphical user interface 10 is shown with an application action area 20 includes an icon 21 for selecting an action such as retrieving, downloading, or otherwise accessing documents, photos, etc.; selecting an emoji, such as a heart 22, an area for writing a message 23, and a send icon, here arrow 24, to transmit the retrieved, downloaded or otherwise accessed documents, phots, etc., and the message and emoji, accordingly by selecting the send icon, an arrow in this example 24. At the same time a settings icon 30 and the user icon 40 are displayed, and together with the action area 20, occupy less than 10% of the display area of the graphical user interface 10. When the user selects the settings icon 30, a corresponding screen for settings appears, and is maintained displayed on the graphical user interface 10 until the user completes entering one or more settings or selects an icon to close the settings screen on the graphical user interface.

[0008] In order to perform an action, the application uses the user hand gestures while they are touching the screen, voice commands, or the users hand gestures or tracking of the user’s eye using a secondary camera facing the user, to create actions based on the application functionality and hardware capabilities. The PIA actions or interactions are activated by dragging an icon corresponding to an action or interaction to the location of the identified object or person on the graphical user interface 10. Once this action is completed the icon could remain displayed as attached to the identified object/person on the graphical user interface 10 or be removed based on the user settings. In the event of multiple objects present in the viewable graphical user interface, the PIA allows the user to select the object that needs to be identified and confirmed by the platform as the“identified object”.

[0009] In general, by pointing the DCD digital camera toward a body or object with whom or which the user is interacting or desiring to interact, an image recognition routine is activated by the platform. This image recognition could be either a face recognition program or routine configured to confirm the identity of the person the interaction is or will be taking place with, or to identify an object such as a such as a business store front, an airline check-in counter, a billboard, a pet such as a dog, or any other living or inanimate object, etc., such image recognition being known in the art and available to the portable DCD either as originally configured, or by an app available thereto from an App store. The objects are then recognized and converted into digital end points with interactables that the PIA can communicate with using the platform’s own application programing interfaces (APIs) and frameworks. In addition, the platform maintains the security of the images, specifically human images by its own algorithms and encryption originally configured thereinto by the DCD manufacturer or added thereto from an App store or other source, as is known in the art, that ensure only images authorized by the user are accessible. Furthermore, if the DCD is operating using one or more location services, such as ones enabled by global positioning devices or cell towers, the automatic identification image enables the user to perform real time actions within or near a neighborhood or area of the identified object via a geo fence based on data managed by the platform and application settings. If the object is a business or other publicly accessible locale which the user has entered, the PIA allows for real time interaction with the locale such as with locale service providers, and or locale websites, which connect to the platform through applications originally configured into the DCD by the manufacturer thereof, or available from an App store, the locales’ service providers, and or locales’ websites its employees, and the user peers, friends, family, associates, etc. that are present at the locale.

[0010] The PIA, through the platform application programming interfaces (APIs) interacts by downloading or just communicating with on a temporary basis, any support application that is necessary to complete any action with no- or minimum- changes of screens and within the PIA using an internet connection, a locale intranet connection, peer to peer wireless or wired communication, or other communication protocol. Once the temporary interaction is completed, the support application is closed. If the PIA detects that a support application does not exist within the DCD, it prompts the user to download and install such application from the corresponding application store, all within the PIA environment. Furthermore, if the user selects to not download the missing support application, the PIA requests from the platform providing the digital information to the PIA to host the temporary application on a temporary basis to access and operate the application or locale based support applications as will be described further herein, but any related information used by the PIA can be erased per user selectable settings. As a result, the user’s smart device (DCD) does not become overloaded with application software and data for applications that cannot be operated when the user is in a different location. Similarly, apps can be downloaded and loaded by another trigger, such as a user action or activity being detected by the smart device, for example, the user entering a locale(s), within which the device had interacted previously and where the user has previously authorized the DCD to receive applications downloaded thereinto from the locale when within the geofence, i.e., within a previously entered geo fence.

[0011] Localized General Interactions:

[0012] In reference to Figure 2a, while the user is outside a locale’s geo fence, but within view of the locale, and the corresponding image of an object detected by the DCD camera is identified as indicated, for example, by a frame that appears around the object, Figure 2b, and the user confirms the selection by touching the screen within the frame, Figure 2c, the PIA offers the user a corresponding set of actions. For example, if the locale corresponding to the image is a restaurant, the PIA offers to call the restaurant or place an online reservation therewith, without the need to exit the PIA. Similarly, the PIA offers the user the option of ordering concert and movie tickets, etc. if the object imaged by the DCD camera is an entertainment venue. If the object imaged by the DCD camera is a billboard or other advertisement media, the PIA offers the user a link to the corresponding website for the information shown, or to what the advertiser has set in the web as a link, which could display promotions, discount coupons, etc. related to the advertiser or a third party. If the image is a car connected to a ride sharing hailing service or a bicycle connected to a public bike use program, the user can reserve the mode of transportation in the imaged object. It is obvious to someone skilled in the art that the image identification by the PIA/platform can be confirmed to the user by many other means, such as the icon blinking, changing color, icons, etc. Furthermore, it is obvious to someone skilled in the art that many other examples of digital interactions with physical objects are possible.

[0013] If the image of a locale is identified and confirmed, and the user is within the locale, then the platform or PIA enables the operation of the application, and by extension its user, to digitally interact with the locale, and the PIA offers actions possible accordingly. For example, at a restaurant, the restaurants’ online webpage can be displayed offering the menu thereof. When a waitperson comes to the user’s table in the restaurant, and the camera is pointed at him or her, he is automatically identified as an employee of the locale by previously tagged or provided data by the locale enabling an interaction by which a food order can be placed, and at the end of a meal, when the bill is presented, process a payment based on any of the digital payment systems available or becoming available such as ApplePay, Square, or other digital payment service. Similarly, a food, drink or food and drink order could be placed directly to the restaurant online order system, if available, bypassing the waitperson.

[0014] Localized Personal Interactions:

[0015] In reference to Figure 3a, once an image of the person with whom the user is interacting in a locale is detected by the platform and the identification of the identity of the person is confirmed by the PIA or platform to be within the known circle of known peers, friends, relatives, etc. of the user, the PIA or platform causes it to be displayed in a frame around the face of the person, or using another indication viewable by the user, as shown in Figure 3b,. Once this occurs, any digital possible personal interaction with that person through the PIA or platform is enabled. Additionally, where the OCD camera detects an image of an individual other than one identified as being an employee or contractor of the locale, such as a of known peers, friends, relatives, etc. of the user, the PIA can send the image captured by the DCD camera to the platform which identifies the imaged other user and sends back user information to the PIA, and simultaneously sends the other user a message to turn on their PIA if installed in his/her DCD or a text message to invite the other user to download the PIA. It is obvious to someone skilled in the art that the image identification by the PIA/platform can be confirmed to the user by many other means, such as a blinking light or screen on the other users’ device, change in lighting or pose etc. on their device, If the person that the user is trying to interact does not have the interactive application installed in his/her DCD, and if this occurs, the other user can request from the individual from whoms’ device the request to communicate was received, an ability to download and install the relevant PIA. However, the platform, if enabled by the user of an application, could enable any individual to digitally communicate with the user or block the user.

[0016] The type of personalized interactions includes document sharing, emoji’s, text messages, transferring cash between the users in the interaction, and any other digital interaction. In the case of monetary transfer or payments, when multiple users are at restaurant and sharing the expense of a meal, for example, one user can take care of the payment for the meal, and then by pointing the device to each of the other users at the table, receive from each user the transfer to the one user of the other user’s corresponding portion of the bill. This personal interaction eliminates the need to handle cash or multiple credit cards to make a payment for a joint expense.

[0017] This method of communication can also be used for submission of assignments at schools, colleges, etc.; where the user points the DCD camera to the class teaching assistant or the teacher, ensuring not only delivery, but also confirmation that the assignment is being delivered by the student on time and to the right recipient, as simultaneously the PIA confirms to the student the delivery of the assignment. Another example of an interaction is one between individuals present at an event within a locale’s geo fence. Event organizers of an event invite participants to register to the event by including a digital image of themselves and general information that includes in addition to their name and affiliation, areas of interest, thus enabling an effective way for participants at an event to identify people with common interest and facilitate identification of whom to approach by image/information identification prior to verbal introduction.

[0018] Herein, Figure 4 to 1 1 outline the interaction of a user device, such as a smart device, and a PIA thereon, with the user environment to select and download locale specific or other applications (“Apps”) for interactive communication through the users’ device. This is also disclosed in applicants’ U.S. Patent Application Serial No. 15/893,198, filed February 9, 2018, which is incorporated herein by reference. In operation, the users’ device can be preprogrammed by the user to authorize acceptance of application data from a specifically identified application program which can be downloaded to the user’s device as an App, where the application program is available wirelessly only within a specific geo fence. When the users’ device is in the proximity (within the geo fence) of a specific location where an application is available by wireless or cellular broadcast, the application, if authorized to be accepted by the user as a specifically identified application, will be automatically downloaded and displayed on the users’ device as a user selectable icon or by other user selection capabilities. The geo-fenced area can be specified by the developer of an application or determined based on the amount of data needed to be transmitted/downloaded/installed, or, the user can specify, prior to arriving at the location, the possible area within the geo fence in which a specifically identified application may be downloaded to their device. This downloaded application can be permanently stored on the device into which it was downloaded, or uninstalled at an instance specified by the user or by the provider. For example, the user can uninstall from their device the app which allowed the provider to automatically download the local content useful in within the geo-fence area to the users’ device, thereby preventing future downloading of the specific application unless the app is again installed, modify the app settings to disallow the application from being downloaded thereby preventing future downloading of the specific application, establish settings to erase the application based on a change in location of the users’ device, or manually erase the application. Optionally, where the application is erased, it may be revived, if not yet overridden in the device memory, when the users’ device reenters the geo-fence or the limited geo-fence region specified by the user, thereby not requiring re-downloading thereof from the specific location’s wireless or cellular system. Such an instance could be based on a specific time or specified passing of time, specific location, of the user device, a combination of both, or determined by the device system or developer. However, the user devices’ operating system or the application developer can preserve the user-specific application account data for future use, preferably in the operating system of the users’ device. In addition to specific locations, apps can be assigned to specific objects that can be identified around a person. For example, if a person is sitting at a desk, certain applications can be assigned to the desk, but also other desks in other rooms, or to other users’ devices which will not be location specific. Similarly, apps can be downloaded and loaded by another trigger, such as a user action or activity being detected by the users’ device, for example, detecting when a person begins or stops running, etc. [0019] Referring to Fig. 4, an example of the operation of the users’ device, including the processer, memory, and communication device thereof, is shown. In act 100, a user carrying a users’ device, and thus the device itself, enters into a geofenced application download trigger area, where the local area network (LAN) is broadcasting availability of an application useful within the boundaries of the geofence. The users’ device determines whether the application has been authorized, by for example the user thereof by authorizing downloading of the application on the screen of the device when the availability of the application is displayed, or by a previous authorization, to be downloaded thereto, act 102. If the downloading of the application is not authorized, No in Fig.4, the users’ device does not download the application. If the downloading of the application is authorized, the users’ device determines whether the application, i.e., the application data, is already stored in the memory of the users’ device, for example the cache memory thereof, and if not, proceeds to download the application data thereto in Act 106. If the application data was in the memory of the users’ device, or if not, once it is downloaded to the users’ device, the users’ device then determines whether the users’ device is within a designated area of the geofence for use of the application in Act 108, and if so, loads and runs the application in Act 1 10. If the user has not entered the designated area in the geofence for use of the application, No in act 108, the users’ device again determines whether the user has entered the designated area for use of the application in Act 108. Repeating of Act 108 may be based on the passage of time from the last execution of Act 108, or the receipt of a trigger signal from the LAN that the appropriate area within the geofence for use of the application has been entered. Maintenance, i.e., continued storage, of the application and the attendant data on the users’ device are preferably prioritized, whereby applications and data can be assigned to at least three categories: Maintained unless manually erased, always erase, i.e., allow the memory in which they are stored to be overwritten, and maintain if the memory space in which they are stored is not needed to optimally, as defined by the user, operate. In each category, the user or operating system may specify the application maintenance priorities. In the third category, the applications and data may be stored in an over- writable cache, wherein the user or operating system can set the priority for overwriting. For example, where an application is always operated within a geo- fence where the application is provided over a specific location provider local transmitter such that no cellular service is involved, the user is likely to specify that the application be“erased”. On the other hand, applications which the user is likely to reuse and which are also received by a cellular connection will be likely moved to the overwritable cache. The over-writable memory, such as a cache memory, can be configured to overwrite the oldest data first. Thus, where a user inadvertently, or purposely, moves in and out of the geo-fence within which the application is accessible in a short period of time, the users’ device will likely still have the application and most recent data therein still available in cache, from which it can be retrieved upon reentering the geo-fence.

[0020] By having a trigger such as an object, action, or location cause an application to install and run, the user device does not need to store extensive application data and can allow for other applications to use the data storage space of the users’ device. If necessary, only user Meta data is stored at the user’s own choice or stored at the service provider, if such a capability is offered and at the user’s own selection. In addition, the trigger decreases the number of interactions a user needs to download a relevant app. In this case, the application will be downloaded because it is deemed relevant to the user by the trigger.

[0021] Referring to Fig. 5, once the user device leaves the application geofence, Act 200, if the device has not been authorized by the user to maintain the application in memory, the users’ device removes the application from the from the foreground, Act 202, such as by removing an icon relating thereto from the device screen, and then sets the application into the cache memory by organizing thereinto based on priority of potential reuse. Act 204. Then, during use of the users’ device, the users’ device determines whether device data to be stored, or the use of the device, requires that data in the cache needs to be overwritten, ACT 206. If it does, YES in Act 206, the data in the cache is overwritten, wherein the data having the lowest priority is overwritten first.is allowed to be overwritten.

[0022] Thus, by organizing downloaded applications (apps), including those useful only is a specific locale, in the users’ device memory based on a user designated priority, the operation speed and capability of the users’ device can be maintained despite the user using multiple geofence specific applications over time. Additionally, by setting the overwriting, or the maintenance, priority based on the users perceived need to again use the application, frequency of use of the application, or likely order in which downloaded applications will again be used, the user of the device will perceive a minimum impact on their use of the device. In many cases, depending on the device memory capacity, and the frequency of downloading of, and the memory required for, different the downloaded applications, the application frequently used will have an overwrite status in the device memory, but, because the overwrite priority assigned to the application is low, or the maintenance priority to maintain the application is high, it will remain in memory and will be available as soon as the user enters the geofence therefor, and the time required to download and execute the application on the device is eliminated, resulting in enhanced user experience.

[0023] To address the issue of images of an individual wearing a user device such as an augmented reality headset including the image of the users’ device, a framework is provided herein for users’ device owner to be viewed in media, i.e., in augmented reality on another users’ device and in real time as if a device that is blocking part of the person’s body from view is removed from view. Using computer vision recognition, a viewing device can detect when an obstruction is in the view of the viewing device, or can also be informed by an outside source such as the obstruction, for example an obstructing users’ device or an external communication platform that communicates therewith, that there is a device obstructing the view of a person’s body within an image. The external communication platform can coordinate between the two users’ devices based on relative location and field of view of the user device to the person being viewed. If the viewing device’s user has allowed for the artificial removal of the obstructing device from the image, the viewing device will request access to visual data about the owner of the body part obstructing users’ device. The viewing device can request this information directly from the obstructing device or from a communication platform on an external server. If the obstructing device’s user has allowed for the transfer of the visual data, the data is sent to the viewing device from the obstructing device or from the external communication server. The viewing device then uses the visual data to overlay and blend an image of the obstructed portion of the obstructing device user’s body onto the area where the obstructing device is present and obstructing and into the augmented reality or other scene being viewed by the viewing user.

[0024] The visual data transmitted includes a general 3D model of the owner of the obstructing device. This data can be enhanced by additional information from the obstructing device and the viewing device. If the obstructing device is, for example, covering part or all of the user thereof s face, and it has access to the pupil color and gaze direction of that user’s eyes, that information can also be included in the 3D model. If the obstructing device has the proper sensors onboard, facial structure such as smiling and frowning can be added to the 3D model. Additionally, environment conditions such as lighting can be used to blend the 3D model into the viewer’s scene. If the additional data is not accessible, the data can be generated using artificial intelligence and computer graphics which adapt based on obstructing device owner’s information.

[0025] Once the information is received it is overlaid over a generic model where animation and natural behaviors are embedded enabling a view of the obstructed part of the other user with its expression and movement.

[0026] The framework can be implemented for still images, moving video, live media, or other media.

[0027] For example, a token or other identification mechanism for the person being viewed can be present on the user, or for example, in the users’ device obstructing the body part. This token can be sensed by the users’ device through which the obstructed user is viewed, or through a wireless LAN or other detection system connected to a storage holding an image of the individual owner of the token, which can identify the person having the body part obstructed. In either case, the image information is then used to generate, in the image of the individual having the body part blocked viewed by the user.

[0028] Referring to Fig. 6, when a user’s device detects a person wherein a users’ device is blocking a part of their body, ACT 300, the users device determines whether the device blocking a part of the body of the other user allows for virtual removal of the obstructing device by the users device, ACT 102. If NO, this processing stops. If yes, in a\ACT 304, the users device determines if it has access to the owner of the obstructing device’s visual data, i.e., an image of their face or the portion thereof blocked by the device, ACT 304. If NO, this processing ends. If yes, the user’s device displays the image of the owner over the obstructing device in the image it displays.

[0029] By overlaying a person’s body part onto their device in pictures or through a display, the image of the person wearing the device will look more natural to others viewing them through another device and this preserves social conventions, such as those occurring during conversation, without the need to use and/or transmit large amounts of data to one or both of the obstructed users device or the viewing users device. This will further facilitate the use of naturally looking avatars in computer driven real time interactions.

[0030] To address limitations in current processing power of portable devices being used to display camera/lens images, and simultaneously process additional information such as rendering of augmented reality content on the same image, a system is provided which ameliorates the current limitations of computational power in a device by learning the tendencies and status of a users’ device user and distributing computing power based on that collective information. The system collects data about the areas in a view where an individual is looking. This information includes what object is being viewed, and which parts of the object are being focused on at the moment, as well as over time, as well as details about the object such as color and orientation. The system can also simultaneously collect data about the individual’s status such as their physical activity, mental state, eyelid status, and schedule/notifications. The system collects such data based on motion sensors, eye trackers, cameras, and computer vision technologies on the users’ device. The combination of data is used to build a library of what the individual focuses on in a view, how the focus moves based on user status as well as on the user field of view the or that in the fovial view in the user field of view.

[0031] The system can use the library of data and current data about the individual to predict the fovial view of a user in different digital images, what specific subsections of the digital image will be in the fovial view, and how the fovial view, i.e. , the direction of the eye, will move. The prediction can be used to allocate computational power to render the areas of the digital images in which the fovial view of the individual is predicted to be directed, thereby dedicating more of the limited computational resources of the device to the expected area of the image to be in the fovial view and thereby creating a more detailed, a more frequently updated, or a more detailed and more frequently updated, image in that area. The system is tailored to the individual user of the device on which the images are displayed, to learn the viewing preferences of the individual and to be able to dynamically react to the individual using a prediction of the next area of the display to be in the fovial gaze of the user.

[0032] By predicting where the foveal gaze and attention of the user will be with respect to the display or screen, and the importance of the quality of the images to be displayed, the processor can prioritize where to increase and decrease the quality of an image, 3D model, or image and 3D model generated and rendered, thus reducing the amount of required computational power and data density required to display an image to a user which appears more natural in the area of the image the user is viewing, is next predicted to view, or is viewing and is next predicted to view.

[0033] Referring to Fig. 7, when an object or image enters a person’s field of view, ACT 500, the gaze tracker matches the position of a user’s eye to the position of an object or person to determine where or what the user is viewing, ACT 502. Then the processer of the system records what the user is looking at, and where they are looking, ACT 504. The processer, based on the record of what the user is looking at, and where they are looking, determines whether graphic images should be renders and added to the users view, ACT 506. If a determination is made not to render graphical images, the processer returns to the result of the ACT 504. If a determination to render graphical images is made, then the processer determines whether the system has a prediction of where the fovial gaze of the user will focus, ACT 508. If no prediction is present, images are rendered, ACT 510. If a prediction is present, the processes allocated processor power, i.e., computing power, to render images in the order of where the system predicts the user will gaze. Herein, a framework that provides an accessible ability to capture, maintain, and then use/view a 2D and 3D image is provided. Using computer vision and recognition, the capturing users’ device can assign a general 3D mesh to the object that is being scanned. As the capturing device views the scanned object, it adds 2D and/or 3D image data to the corresponding locations of the mesh. The 3D mesh specific to the scanned object can then be stored on the capturing device or on an external server. When another capturing device scans the scanned object subsequent times after the initial scanning, it applies the additional data to the existing mesh and updates mesh to increase accuracy and status of the object.

[0034] The 3D mesh can then be played back in 2D or 3D still, moving, and real time media. As the scanned object is recorded changing position or appearance, the mesh can be adjusted to replicate the position/appearance in real time. Applying captures to a generalized mesh of an object, the subject can be scanned and captured in 3D with limited capturing hardware and time.

[0035] Referring to FIG. 8, a capturing device, such as a graphics processer receiving images captured by a camera, first reviews objects and receives inputs, and determines if an object captured by the camera or of which the processer has been informed can be represented as a 3D scale model, Act 600. If so, the processer determines whether the object has been previously assigned a 3D mesh, ACT 602, and if not, assign a 3D mesh to the object that most closely matches the shape of the object, ACT 604. If a 3D mesh was previously assigned to the object, the processer captures the image of the object in a memory, and adds the image to the matching location of the 3D mesh and blends it into the mesh, ACT 606

[0036] Herein a 3D digital interface for viewing, selecting, and making online purchases is provided. The interface constitutes a physical dimension in real space and includes 3D scale representations of the product offerings that a shopper can view from different angles and interact with. The shape of the interface can constitute different styles of shops such as coat racks, kiosks, shelves, etc. The shape and design of the “store” is determined by the retailer and the physical restrictions of the space of the buyer. The product offering and scale is also determined by the retailer. [0037] A 3D store can match the experience of a physical retail store, but do so in spaces where the shopper is rather than where the store is.

[0038] Referring now to Fig. 9, a schematic of the processing portion of a device hereof is shown. Device 10 includes a microelectronic processor 700 for performing arithmetic calculations and receiving external inputs on input terminal 702, and generating external outputs over output terminal 704. Processer 700 is connected to a read only memory (ROM) 708, a random access memory (RAM) 712, an antenna 712 and a display 714.

[0039] In use, operating software for the operation of the functions of the processer 700 are stored in the ROM 708, and programs used intermittently, such as those downloaded to the device 10 via the antenna 712, and memory required to store data generated by these programs, is stored in the RAM after being received by the processor through the antenna. The processor, which includes a graphics processor integrally or separately provided, therewith, renders images on the display

[0040] Figure 10 is a schematic representation of a the viewable portion of a users’ device 10. Device 10 includes a casing 800, a screen 802, an on/off switch 804 and here at least one user selectable icon 806 thereon.

[0041] It is obvious to someone skilled in the art that other use cases/interactions between a user of DCDs and his/her peers and the environment are possible and follow the same general design and methodology of the platform and PIA described here.

Claims

What is claimed is:

1. A platform residing in the cloud, a physical server, desk-top computer, or digital mobile devices which recognizes images of objects and humans and converts them to a digital end points for the purpose of interactions between users of applications on digital communication devices.

2. The platform of claim 1 where the images can be any living and non-living object in the environment and or surroundings of the user operating a digital communications device.

3. The platform of claim 1 where the images of claim 2 are either human’s or animal’s faces.

4. The platform of claim 1 where the image recognition includes either, or both, locale name and geo positioning.

5. The platform of claim 1 which includes APIs to websites, mobile applications, and other computing applications to enable performance of functions in such applications.

6. The platform of claim one, where a geo fence is created for locale based activities by applications connected to the platform.

7. An application for digital communication devices which includes a graphical user interface that enables viewing images obtained via a camera, integrated to- or connected to-, the digital communication device; in at least 30% of its area, while the rest of the area is occupied by icons necessary to execute the application’s functions.

8. The application of claim 7, where the graphical user interface views images occupying at least 60% of the viewing area, while the rest of the area is occupied by icons necessary to execute the application’s functions.

9. The application of claim 7, where the graphical user interface views images occupying at least 85% of the viewing area, while the rest of the area is occupied by icons necessary to execute the application’s functions.

10. The application of claim 7, where additional functions are displayed in a temporary base.

11 . The application of claim 7, which interacts with the platform of claim 1 to identify and confirm the identity of images.

12. The application of claim 7 which allows the user to select the object that needs to be identified and confirmed by the platform, especially in the presence of multiple objects such as persons, buildings, etc.

13. The application of claim 7, where interactions enabled by the application allow for the user to have a direct viewing field including the surroundings of the object with whom the application user is interacting.

14. The application of claim 7, where the interactions include as an example a real time exchange of messages, documents, payments, emoji’s, etc.

15. The application of claim 7 where interactions take place by dragging an available and selected icon to the confirmed image by means of the user’s hand gestures touching the screen, voice commands, or using a secondary camera facing the user; hand gestures waving or tracking the user’s eye to create actions based on the application functionality and hardware capabilities.

16. The platform of claim 1 which enables users of applications connected to it to interact with identified objects within a predefined geo fence.

17. The platform of claim 1 that hosts temporary applications required by the user application on its digital communication device but that the user is not downloading into the device.

18. The platform of claim 1 that keeps track of temporary applications and switches to permanently hosting and maintaining updated applications commonly used by the users in a temporary base on their digital communication devices.