JP2024025997A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To abstract an object that a user does not want to see in a view field image of a goggle-type wearable device.

SOLUTION: An information processing device according to the present invention comprises: an acquisition unit that acquires a visual field image of a goggle-type wearable device; an identification unit that identifies an object that a user cannot permit from the visual field image; a conversion unit that converts the object into an abstracted content that can be permitted by the user; and a display control unit that displays the abstracted content at a position of the object in a superimposed manner in the visual field image of the goggle-type wearable device.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

A technique has been disclosed that prioritizes and displays virtual objects that the user wants to see.

JP 2021-144258 Publication

However, although the above-mentioned conventional technology refers to displaying virtual objects that the user wants to present, it does not mention displaying real objects or virtual objects that the user does not want to present. For example, in a glasses-type wearable device that has an AR (Augmented Reality) function, the visual field image becomes the user's field of view, and the distance between the user and the screen is close, so the user tends to look away. They are unable to look away and have no choice but to look at what is displayed in the visual field image, even if they do not want to see it. However, if something that the user does not want to see is simply hidden or made invisible, the user's visibility and recognition will be impaired.

The present application has been made in view of the above, and aims to abstract what the user does not want to see in the visual field image of a glasses-type wearable device.

The information processing device according to the present application includes: an acquisition unit that acquires a visual field image of a glasses-type wearable device; a specifying unit that identifies objects that are unacceptable to the user from among the visual field images; and a display control unit that displays the abstracted content in a superimposed manner at the position of the object in the field of view image of the glasses-type wearable device.

According to one aspect of the embodiment, it is possible to abstract what the user does not want to see in the visual field image of the glasses-type wearable device.

FIG. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to an embodiment. FIG. 3 is a diagram illustrating a configuration example of a terminal device according to an embodiment. FIG. 4 is a diagram showing an example of an abstract information database. FIG. 5 is a diagram illustrating a configuration example of a server device according to an embodiment. FIG. 6 is a diagram showing an example of a user information database. FIG. 7 is a diagram showing an example of a history information database. FIG. 8 is a diagram showing an example of a model information database. FIG. 9 is a flowchart showing the processing procedure according to the embodiment. FIG. 10 is a diagram showing an example of a hardware configuration.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. In addition, in the following embodiments, the same parts are given the same reference numerals, and redundant explanations will be omitted.

[1. Overview of information processing method]
First, with reference to FIG. 1, an overview of an information processing method performed by an information processing apparatus according to an embodiment will be described. FIG. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. In addition, in FIG. 1, a case will be described as an example in which what the user does not want to see in the visual field image of a glasses-type wearable device is abstracted.

As shown in FIG. 1, the information processing system 1 includes a terminal device 10, a server device 100, and AR goggles 200 (AR glasses). The terminal device 10, the server device 100, and the AR goggles 200 are connected to be able to communicate with each other by wire or wirelessly via a network N (see FIG. 2). In this embodiment, the terminal device 10 and the AR goggles 200 cooperate with the server device 100. The terminal device 10 and the AR goggles 200 cooperate with each other. The AR goggles 200 may be able to communicate with the server device 100 via the terminal device 10.

The terminal device 10 is a smart device such as a smartphone or a tablet terminal used by a user U (user), and is connected to an arbitrary server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). This is a mobile terminal device that can communicate. Further, the terminal device 10 has a screen such as a liquid crystal display, which has a touch panel function, and displays data such as content through a tap operation, a slide operation, a scroll operation, etc. from the user U using a finger or a stylus. Accepts various operations on. Note that an operation performed on an area of the screen where content is displayed may be an operation on the content. Further, the terminal device 10 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

The server device 100 cooperates with the terminal device 10 of each user U, and provides API (Application Programming Interface) services for various applications (hereinafter referred to as apps) and various data to the terminal device 10 of each user U. It is an information processing device that provides information, and is realized using computers, cloud systems, etc.

Further, the server device 100 may be an information processing device that provides some kind of web service online to the terminal device 10 of each user U. For example, the server device 100 provides web services such as Internet connection, search service, SNS (Social Networking Service), electronic commerce (EC), electronic payment, online games, online banking, online trading, accommodation/ticket reservation, Services such as video/music distribution, news, maps, route searches, route guidance, route information, operation information, weather forecasts, etc. may be provided. In reality, the server device 100 may cooperate with various servers that provide the above-mentioned Web services, mediate the Web services, or may be in charge of processing the Web services.

Note that the server device 100 can acquire user information regarding the user U. For example, the server device 100 acquires information regarding user U's attributes such as user U's gender, age, and residential area. The server device 100 stores and manages identification information indicating the user U (user ID, etc.) as well as information regarding attributes of the user U.

Further, the server device 100 acquires various kinds of history information (log data) indicating the actions of the user U from the terminal device 10 of the user U or from various servers based on the user ID or the like. For example, the server device 100 acquires a location history, which is a history of user U's location, date and time, from the terminal device 10. The server device 100 also acquires a search history, which is a history of search queries input by the user U, from a search server (search engine). Additionally, the server device 100 acquires a viewing history, which is a history of content viewed by the user U, from the content server. The server device 100 also acquires a purchase history (payment history) that is a history of product purchases and payment processing by the user U from the electronic commerce server and the payment processing server. Further, the server device 100 may acquire the listing history and the sales history, which are the listing history of the user U on the marketplace, from the electronic commerce server or the payment processing server. The server device 100 also acquires a posting history, which is a history of postings by the user U, from a posting server or SNS server that provides a word-of-mouth posting service. Note that the various servers described above may be the server device 100 itself. That is, the server device 100 may function as the various servers described above.

The AR goggles 200 (AR glasses) are glasses-type wearable devices having an AR (Augmented Reality) function. Therefore, the AR goggles 200 have a camera function and a short-range communication function. Furthermore, it may have an audio function, a microphone function, an image recognition function, a voice recognition function, etc. The AR goggles 200 can also be used in virtual spaces and metaverses. Note that in reality, the AR goggles 200 are not limited to the AR function, and may have a VR (Virtual Reality) function or an MR (Mixed Reality) function. Further, the AR goggles 200 may be smart glasses equipped with an AR function, an AR/VR headset, a head mounted display (HMD) that serves as both VR goggles and AR goggles, or the like. Further, the AR goggles 200 may be not only a device with a non-transparent display, but also a see-through type AR device such as a hololens.

[1-1. Abstraction AR]
In this embodiment, user U (user) is wearing AR goggles 200. The AR goggles 200 captures field-of-view images with a mounted camera. At this time, a predetermined object that the user U cannot tolerate/does not want to see may be displayed in the visual field image. When the user U is wearing the AR goggles 200, the field of view image becomes the user U's field of view, and the distance between the user U and the screen is close, so the user U is unable to look away or look away. They cannot help but see what is displayed in the visual field image even if they do not want to see it.

However, if the object is completely erased (made invisible) from the visual field image of the AR goggles 200, the object may actually exist in front of the user's eyes, but only the user U cannot see it. In some cases, undesirable situations may occur. In addition, if you simply mosaic (conceal) the object in the visual field image, if the object to be mosaiced is not uniquely identified, you will not know why it was mosaiced or what is actually there. Since the user U does not know whether there is a person or not, there is a lack of information that the user U can obtain from the visual field image.

In this embodiment, the terminal device 10 of the user U (user) cooperates with the AR goggles 200 to acquire a visual field image of the AR goggles 200 (a visual field image of the user U wearing the AR goggles 200). A predetermined object OBJ is detected from the visual field image, and abstracted content ABS (abstracted icon, etc.) that is an abstraction of the detected predetermined object OBJ is generated when the predetermined object OBJ in the visual field image is located. Display superimposed on the position.

For example, as shown in FIG. 1, the terminal device 10 of the user U (user) acquires a visual field image taken by the camera function of the AR goggles 200 by short-range wireless communication (or via a communication cable). (Step S1).

Subsequently, the terminal device 10 of the user U identifies and extracts a predetermined object OBJ set in advance from the acquired visual field image (step S2).

Examples of the predetermined object OBJ include things that the user U does not want to look at directly, things that cause disgust, things that cause mental pain, and the like. Note that the setting of the predetermined object OBJ may be set, for example, by initial setting at the time of installing the application, by automatic update by the application or software, or by manual setting by the user U using the terminal device 10. You can also set it with Furthermore, the terminal device 10 estimates a predetermined object OBJ for each user using machine learning or the like based on the user attributes of the user U (user segments or user personas are also possible), past history, current settings, etc. It's okay.

At this time, the terminal device 10 of the user U may specify and extract the object OBJ for each category. Categories of object OBJ include, for example, insects/animals, garbage/dust, pictures/photos/videos, characters/text, expressions based on specific ideas/beliefs, and the like. Further, the category of the object OBJ may be subdivided into major classification, medium classification, and small classification. For example, the major classification may be "organisms," the medium classification may be "insects," and the minor classification may be "types and names of insects."

Subsequently, the terminal device 10 of the user U determines the tolerance level of the user U with respect to the predetermined object OBJ and the degree of abstraction when abstracting the object OBJ (step S3).

At this time, the terminal device 10 may manually accept the settings of the predetermined object OBJ as well as the tolerance level and abstraction level from the user U, or the user U's user attributes, past history, and current settings. The tolerance level and abstraction level may be estimated for each user based on the settings and the like using machine learning or the like.

Subsequently, in order to abstract the predetermined object OBJ according to the tolerance level and abstraction level of each user U, the terminal device 10 of the user U generates an abstracted content ABS that abstracts the predetermined object OBJ. is selected or generated (step S4).

At this time, the terminal device 10 may abstract the predetermined object OBJ in stages according to the tolerance and abstraction level of each user U. For example, the terminal device 10 stores in advance a set of a predetermined object OBJ and abstracted content ABS for each tolerance level and abstraction level, and corresponds to the predetermined object OBJ according to the tolerance level and abstraction level. The abstracted content ABS may be selected, or the abstracted content ABS may be automatically generated from a predetermined object OBJ according to the tolerance level and abstraction level. Further, the terminal device 10 may gradually blur the predetermined object OBJ or change the transmittance/transparency depending on the tolerance level or abstraction level.

Subsequently, the terminal device 10 of the user U converts a predetermined object OBJ included in the visual field image of the AR goggles 200 into abstracted content ABS (step S5).

At this time, the terminal device 10 provides the AR goggles 200 with information regarding the abstracted content ABS, which is an abstraction of the predetermined object OBJ, by short-range wireless communication (or via a communication cable). For example, the terminal device 10 provides information regarding the correspondence between the predetermined object OBJ and the abstracted content ABS, the display position and display instructions of the abstracted content ABS. Note that when the abstracted content ABS is displayed in a special display mode different from normal, the terminal device 10 also provides information regarding the display mode.

Next, the AR goggles 200 displays abstracted content ABS, which is an abstraction of the predetermined object OBJ, in a superimposed manner at a position where the predetermined object OBJ included in the visual field image is located (step S6).

Note that, in reality, the terminal device 10 of the user U displays abstracted content ABS, which is an abstraction of the predetermined object OBJ, at a position where the predetermined object OBJ included in the visual field image of the AR goggles 200 is located. The display may be controlled to be displayed in a superimposed manner.

Subsequently, the terminal device 10 of the user U cooperates with the server device 100 via the network N (see FIG. 2) and provides the server device 100 with information regarding the abstraction of the predetermined object OBJ (step S7 ).

At this time, the server device 100 collects information regarding the abstraction of the predetermined object OBJ from the terminal devices 10 of an unspecified number of users U via the network N (see FIG. 2). Alternatively, when performing machine learning in a distributed state without aggregating data by federated learning, the server device 100 transmits learning results ( (difference data or feature amounts) may also be acquired.

Next, the server device 100 uses machine learning to learn about the abstraction of a predetermined object OBJ for each user attribute (user segment, user persona) of the user U, and generates an estimation model based on the learning results. Update (step S8).

For example, the server device 100 learns a set of a user attribute, a predetermined object OBJ, a tolerance or an abstraction level, and an abstracted content ABS as learning data, and when the user attribute and a predetermined object OBJ are input, the tolerance An estimation model is generated/updated to output the level of abstraction, abstraction level, and abstracted content ABS. Alternatively, the server device 100 may integrate the learning results obtained from the terminal device 10 of the user U through federated learning, and generate/update the estimated model.

Subsequently, the server device 100 provides the generated/updated estimation model to the terminal device 10 of each user U via the network N (see FIG. 2) (step S9).

Thereby, when identifying and extracting the predetermined object OBJ from the acquired visual field image in step S2, the terminal device 10 of the user U uses the predetermined object OBJ based on the estimation model provided from the server device 100. It becomes possible to estimate the target object OBJ and identify and extract the estimated predetermined target object OBJ.

Note that the server device 100 is not limited to the generated/updated estimation model itself, but also the global model (central model) on the server device 100 side corresponding to the estimation model and the local model (child model) on the terminal device 10 side of the user U. model). For example, when performing machine learning in a distributed state without aggregating data by federated learning, the server device 100 transmits information regarding correction points of the shared model/integrated model on the server device 100 side to the user U's terminal device 10. may be provided to

[1-2. Additional/supplementary matters]
In the present embodiment, the terminal device 10 of the user U changes the predetermined object OBJ included in the visual field image of the AR goggles 200 to the abstracted content ABS. Then, the terminal device 10 of the user U displays the abstracted content ABS, which is an abstraction of the extracted object OBJ, within the visual field image. For example, the terminal device 10 of the user U displays the abstracted content ABS superimposed on the position where the object OBJ is located.

Note that the abstracted content ABS may be a two-dimensional (2D) image, a three-dimensional (3D) image, or a 3D object/3D model (3D shape). Further, the terminal device 10 of the user U may automatically generate the abstracted content ABS from the object OBJ.

Further, when extracting the predetermined object OBJ, the terminal device 10 of the user U may determine whether to extract the predetermined object OBJ according to the size, distance, etc. within the visual field image. Specifically, the terminal device 10 of the user U detects a predetermined object OBJ when the size, distance, etc. are such that the user U can recognize (visually recognize) the predetermined object OBJ as such in the visual field image. The object may be extracted as an object OBJ to be abstracted. For example, the terminal device 10 of the user U detects that a predetermined object OBJ exists within a predetermined range (for example, within a distance of 2 m) from the user U in the visual field image, and has a predetermined size (for example, a total length of 1 cm or more). When displayed, a predetermined object OBJ may be extracted as an object OBJ to be abstracted.

In addition, the terminal device 10 of the user U changes the extracted predetermined object OBJ to a "character" according to the tolerance level and abstraction level, and superimposes the "character" at the position where the object OBJ is located. may be displayed. For example, the terminal device 10 of the user U superimposes the "name" or "category name" of a predetermined extracted object OBJ at the location where the extracted object OBJ is located, depending on the tolerance level and abstraction level. May be displayed.

Alternatively, the terminal device 10 of the user U changes the extracted predetermined object OBJ to an "image indicating a category" according to the tolerance level and abstraction level, and changes the "category" to the position where the object OBJ is located. may be displayed in a superimposed manner. Images indicating categories may be set in advance for each category.

Further, when the extracted predetermined object OBJ is an insect, the terminal device 10 of the user U changes the object OBJ to an "icon/pictogram that abstracts an insect", and changes the location where the object OBJ is located. An "icon/pictogram that abstracts insects" may be superimposed and displayed. Basically, it is assumed that people who are not good with a particular insect are not good with insects in general, so it is preferable to change the image to some kind of pictogram that abstracts the insect, rather than changing it to an image of another insect. At this time, unless all insects are object OBJs, user U's terminal device 10 does not uniformly change all insects to "icons/pictograms that abstract insects", but instead changes them to object object OBJ. You may change only the insect set as OBJ to an "icon/pictogram that abstracts insects" and display the "icon/pictogram that abstracts insects" superimposed on the location where the object OBJ is located. . Furthermore, when displaying an "icon/pictogram that abstracts insects" superimposed on the terminal device 10 of user U, the user U's terminal device 10 displays an image of "an icon/pictogram that abstracts insects" instead of an image of a normal "icon/pictogram that abstracts insects" that exists in the natural world. , an image of an unnatural "icon pictogram of an abstract insect" that clearly shows that it has been changed may be displayed. For example, an ``icon/pictogram that abstracts an insect'' that is several times larger than the normal size, an ``icon/pictogram that abstracts an insect'' with a color or shape that does not exist in the current environment, or an ``icon/pictogram that abstracts an insect'' that emits light or blinks. An icon or pictogram of an abstract insect may be displayed.

Alternatively, the terminal device 10 of the user U changes the extracted predetermined object OBJ to a "caricatured illustration" according to the tolerance level or abstraction level, and changes the extracted predetermined object OBJ to a "caricature illustration" at the location where the object OBJ is located. It is also possible to superimpose and display "illustrations". The caricatured illustrations may be set in advance for each object OBJ or category, or may be automatically converted from the object OBJ by image conversion, machine learning, or the like. Further, the terminal device 10 of the user U changes the extracted predetermined object OBJ to a "pictogram" according to the tolerance level and abstraction level, and superimposes the "pictogram" at the position where the object OBJ is located. may be displayed.

Note that the terminal device 10 of the user U may receive a setting regarding the abstraction level of the abstracted content ABS from the user U regarding the abstraction level when abstracting the object OBJ. For example, the terminal device 10 of the user U may receive a setting from the user U regarding how high the level of abstraction should be increased. Further, the terminal device 10 of the user U may select or automatically generate the abstracted content ABS depending on the level of abstraction.

Further, the terminal device 10 of the user U may inquire of the user U how much he dislikes the object OBJ, and change the abstraction level depending on the answer. Further, the terminal device 10 of the user U may inquire of the user U whether he likes or dislikes the abstracted content ABS, and further changes the degree of abstraction depending on the answer. At this time, the terminal device 10 of the user U may learn the answers from the user U and change the level of abstraction based on the learning results.

Alternatively, the terminal device 10 of the user U may estimate and calculate the abstraction level of the abstracted content ABS and automatically set it. For example, the terminal device 10 of the user U may estimate the preferences of the user U from the user attributes of the user U, and calculate an allowable score for each category according to the estimated preferences. . At this time, the user U's terminal device 10 lowers the level of abstraction as the tolerance score increases.

Further, the terminal device 10 of the user U may estimate the context of the user U and convert the predetermined object OBJ into the abstracted content ABS according to the estimated context. For example, the terminal device 10 of the user U may set or change the abstracted content ABS depending on the context of the user U's location (location) such as at home/in the office/in the city/in a public place/in a car. In addition, the terminal device 10 of the user U can display the abstracted content ABS according to the surrounding context of the user U, such as when the user U is alone (there are no people around) or when there are many people around the user U (there are many people around). may be set or changed.

Alternatively, the terminal device 10 of the user U may operate according to the context of the time (time zone) of the user U, such as coming to work/before going to work/while moving/during working hours/during a meal/during a meeting/leaving/after leaving work. The abstracted content ABS may be set/changed. Further, the terminal device 10 of the user U may set or change the abstracted content ABS depending on the context of the day of the week, such as weekdays, weekends, holidays, etc.

In addition, the terminal device 10 of the user U sets the abstracted content ABS depending on whether the extracted object OBJ is a "real object" or a "virtual object" displayed on the screen by an AR function etc.・You may change it. That is, the object OBJ is not limited to a real object, but may be data displayed using AR (augmented reality) or the like. If the user U really dislikes the object OBJ, the terminal device 10 of the user U changes even a virtual object into the abstracted content ABS. If you don't dislike it that much/depending on the degree of your dislike, the real object will be changed to the abstracted content ABS, and the virtual object will be displayed as is. Whether the user U really dislikes the content may be inquired directly from the user U, or may be inferred from the setting of the abstraction level of the abstracted content ABS.

Note that the above processing of the terminal device 10 may be performed by the server device 100 that cooperates with the terminal device 10, or may be performed by the AR goggles 200. Further, the AR goggles 200 may cooperate with the terminal device 10 via the network N (see FIG. 2), or may communicate with the server device 100 via the network N (see FIG. 2) without going through the terminal device 10. You can also collaborate directly.

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the server device 100 according to the embodiment will be described using FIG. 2. FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes a terminal device 10, a server device 100, and AR goggles 200. These various devices are connected via a network N so that they can communicate by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Furthermore, the number of devices included in the information processing system 1 shown in FIG. 2 is not limited to what is illustrated. For example, in FIG. 2, one terminal device 10 and one AR goggle 200 are shown to simplify the illustration, but this is just an example and is not limited to two or more of each. Good too.

The terminal device 10 is an information processing device used by the user U. For example, the terminal device 10 may be a smart device such as a smartphone or a tablet terminal, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a game console or AV device with a communication function, a car navigation system, a smart watch, etc. It is. Further, the terminal device 10 may be a house/building, a car, a home appliance, an electronic device, etc. compatible with IOT (Internet of Things).

Further, the AR goggles 200 are not limited to AR glasses, and may be glasses-type wearable devices such as smart glasses or a head-mounted display.

Further, the terminal device 10 and the AR goggles 200 are connected to wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation: 5th generation mobile communication system), Bluetooth (registered trademark), It is possible to connect to the network N and communicate with the server device 100 via short-range wireless communication such as a wireless LAN (Local Area Network).

The server device 100 is, for example, a computer such as a PC or a blade server, or a mainframe or a workstation. Note that the server device 100 may be realized by cloud computing.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be explained using FIG. 3. FIG. 3 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 3, the terminal device 10 includes a communication section 11, a display section 12, an input section 13, a positioning section 14, a sensor section 20, a control section 30 (controller), and a storage section 40. Be prepared.

(Communication Department 11)
The communication unit 11 is connected to a network N (see FIG. 2) by wire or wirelessly, and transmits and receives information to and from the server device 100 via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, or the like.

(Display section 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (EL display). Further, the display unit 12 is a touch panel type display, but is not limited to this.

(Input section 13)
The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 includes buttons for inputting characters, numbers, and the like. Note that the input unit 13 may be an input/output port (I/O port), a USB (Universal Serial Bus) port, or the like. Further, when the display section 12 is a touch panel display, a part of the display section 12 functions as the input section 13. Further, the input unit 13 may be a microphone or the like that receives voice input from the user U. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives a signal (radio wave) sent from a GPS (Global Positioning System) satellite, and based on the received signal, determines position information (for example, latitude and longitude). That is, the positioning unit 14 positions the terminal device 10 . Note that GPS is just one example of GNSS (Global Navigation Satellite System).

Further, the positioning unit 14 can measure the position using various methods other than GPS. For example, the positioning unit 14 may use various communication functions of the terminal device 10 to measure the position as an auxiliary positioning means for position correction and the like, as described below.

(Wi-Fi positioning)
For example, the positioning unit 14 positions the terminal device 10 using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication, etc., and determines the position of the terminal device 10 by measuring the distance to nearby base stations and access points.

(Beacon positioning)
Further, the positioning unit 14 may use the Bluetooth (registered trademark) function of the terminal device 10 to measure the position. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting to a beacon transmitter connected by a Bluetooth (registered trademark) function.

(geomagnetic positioning)
Furthermore, the positioning unit 14 positions the terminal device 10 based on the geomagnetic pattern of the structure measured in advance and the geomagnetic sensor included in the terminal device 10 .

(RFID positioning)
Further, for example, if the terminal device 10 has an RFID (Radio Frequency Identification) tag function equivalent to a contactless IC card used at station ticket gates, stores, etc., or has a function to read an RFID tag. In this case, the location where the terminal device 10 used the terminal device 10 is recorded together with the information that the payment was made. The positioning unit 14 may measure the position of the terminal device 10 by acquiring such information. Further, the position may be determined by an optical sensor, an infrared sensor, or the like provided in the terminal device 10.

The positioning unit 14 may position the terminal device 10 using one or a combination of the above-mentioned positioning means, if necessary.

(sensor section 20)
The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. Note that the connection may be a wired connection or a wireless connection. For example, the sensors may be a detection device other than the terminal device 10, such as a wearable device or a wireless device. In the example shown in FIG. 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an atmospheric pressure sensor 23, an air temperature sensor 24, a sound sensor 25, an optical sensor 26, a magnetic sensor 27, and an image sensor ( camera) 28.

Note that each of the sensors 21 to 28 described above is merely an example and is not limited to the above. That is, the sensor section 20 may be configured to include a portion of each of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of each of the sensors 21 to 28. .

The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects the physical movement of the terminal device 10, such as the moving direction, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects physical movements of the terminal device 10 such as tilt in three axes directions based on the angular velocity of the terminal device 10 and the like. The atmospheric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

Since the terminal device 10 is equipped with the above-mentioned acceleration sensor 21, gyro sensor 22, atmospheric pressure sensor 23, etc., it is possible to implement technologies such as pedestrian autonomous navigation (PDR) using these sensors 21 to 23, etc. It becomes possible to measure the position of the terminal device 10 using the . This makes it possible to obtain indoor position information that is difficult to obtain using positioning systems such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. Furthermore, by using the gyro sensor 22, it is possible to know the direction of travel of the user U, the direction of the line of sight, and the inclination of the user's body. Further, from the atmospheric pressure detected by the atmospheric pressure sensor 23, it is also possible to know the altitude and the number of floors where the terminal device 10 of the user U is located.

The temperature sensor 24 detects, for example, the temperature around the terminal device 10. The sound sensor 25 detects, for example, sounds around the terminal device 10. The optical sensor 26 detects the illuminance around the terminal device 10 . The magnetic sensor 27 detects, for example, the earth's magnetism around the terminal device 10. The image sensor 28 captures an image of the surroundings of the terminal device 10.

The above-mentioned atmospheric pressure sensor 23, temperature sensor 24, sound sensor 25, optical sensor 26, and image sensor 28 each detect atmospheric pressure, temperature, sound, and illuminance, and capture images of the surroundings, so that the terminal device 10 It is possible to detect the surrounding environment and situation. Furthermore, it is possible to improve the accuracy of the location information of the terminal device 10 based on the environment and situation around the terminal device 10.

(Control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input/output port, etc., and various circuits. Further, the control unit 30 may be configured with hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control section 30 includes a transmitting section 31, a receiving section 32, and a processing section 33.

(Transmission unit 31)
The transmitting unit 31 receives, for example, various information input by the user U using the input unit 13, various information detected by the sensors 21 to 28 mounted on or connected to the terminal device 10, and information measured by the positioning unit 14. The location information of the terminal device 10 and the like can be transmitted to the server device 100 via the communication unit 11.

(Receiving unit 32)
The receiving unit 32 can receive various information provided from the server device 100 and requests for various information from the server device 100 via the communication unit 11 .

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10, including the display unit 12 and the like. For example, the processing unit 33 can output various information transmitted by the transmitting unit 31 and various information received from the server device 100 by the receiving unit 32 to the display unit 12 for display.

Furthermore, the processing section 33 includes an acquisition section 33A, a setting section 33B, an estimating section 33C, a specifying section 33D, a generating section 33E, a converting section 33F, and a display control section 33G.

(Acquisition unit 33A)
The acquisition unit 33A acquires a visual field image of the glasses-type wearable device. For example, the acquisition unit 33A acquires a visual field image captured by the camera function of the glasses-type wearable device.

(Setting section 33B)
The setting unit 33B accepts settings for objects OBJ that are unacceptable to the user U. Further, the setting unit 33B receives settings regarding the user U's tolerance level for the object OBJ. Further, the setting unit 33B receives settings regarding the level of abstraction when abstracting the object OBJ.

Further, the setting unit 33B inquires of the user U about the degree of liking of the object OBJ, and receives an answer from the user U to the inquiry about the degree of liking of the object OBJ. Further, the setting unit 33B inquires of the user U about the degree of preference for the converted abstract content ABS, and receives a response from the user U to the inquiry regarding the degree of preference for the converted abstract content ABS. Then, the setting unit 33B changes the degree of abstraction when abstracting the object OBJ according to each answer.

(Estimation unit 33C)
The estimation unit 33C estimates an object OBJ that is unacceptable to the user U from various information (user information, history information, etc.) including the user U's attribute information. Furthermore, the estimating unit 33C estimates the user U's tolerance to the object OBJ based on the user U's attribute information and various other information (user information, history information, etc.). Furthermore, the estimation unit 33C estimates the degree of abstraction when abstracting the object OBJ from various information (user information, history information, etc.) including the attribute information of the user U.

(Specific part 33D)
The specifying unit 33D specifies and extracts an object OBJ that is unacceptable to the user U from the visual field image of the glasses-type wearable device.

(Generation unit 33E)
The generation unit 33E automatically generates abstract content ABS from the object OBJ. For example, the generation unit 33E automatically generates the abstracted content ABS from the object OBJ according to the user U's settings regarding the tolerance level for the object OBJ. Further, the generation unit 33E automatically generates the abstracted content ABS from the object OBJ according to the setting regarding the abstraction level when abstracting the object OBJ.

(Converter 33F)
The conversion unit 33F converts the object OBJ into abstracted content ABS acceptable to the user U. For example, when the tolerance level is set, the conversion unit 33F converts the object OBJ into abstracted content ABS that is selected or generated according to the tolerance level. Furthermore, when the degree of abstraction is set, the conversion unit 33F converts the object OBJ into abstracted content ABS selected or generated according to the degree of abstraction.

(Display control unit 33G)
The display control unit 33G displays the abstracted content ABS in a superimposed manner at the position of the object OBJ in the visual field image of the glasses-type wearable device.

At this time, the display control unit 33G may also automatically determine the display mode of the abstracted content ABS according to the user U's settings regarding the tolerance and abstraction level.

(Storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk. Ru. The storage unit 40 stores various programs, various data, and the like.

For example, the storage unit 40 stores a set of object OBJ and abstracted content ABS. The storage unit 40 also stores various settings such as tolerance and abstraction level. As shown in FIG. 4, the storage unit 40 has an abstract information database 40A.

(Abstract information database 40A)
The abstraction information database 40A stores various information regarding abstraction of a predetermined object. FIG. 4 is a diagram showing an example of the abstract information database 40A. In the example shown in FIG. 4, the abstract information database 40A has items such as "object", "tolerance", "abstraction level", "abstract content", and "display mode".

“Object” indicates identification information for identifying a predetermined object OBJ. Further, "tolerance" indicates the user U's tolerance to the predetermined object OBJ. Moreover, "abstraction level" indicates the abstraction level when abstracting the object OBJ. Further, "abstracted content" indicates identification information for identifying abstracted content ABS (abstracted icon, etc.) that is an abstraction of a predetermined object OBJ, or the abstracted content ABS itself (image data, etc.). Furthermore, the “display mode” indicates the display mode of the abstracted content ABS within the visual field image of the AR goggles 200.

For example, in the example shown in FIG. 4, the predetermined object OBJ identified by the object "OBJ#1" is This indicates that the content is converted into abstracted content "ABS #1" and displayed in the visual field image of the AR goggles 200 according to "display mode #1."

Here, in the example shown in FIG. As shown in the figure, "OBJ #1", "Tolerance #1", "Abstract level #1", "ABS #1" and "Display mode #1" contain information such as specific character strings and numerical values. shall be remembered.

Note that the abstract information database 40A is not limited to the above, and may store various information depending on the purpose. For example, the abstract information database 40A may store information regarding the category (classification) of a predetermined object OBJ. Further, the abstraction information database 40A may store information regarding a context when abstracting a predetermined object OBJ. Further, the abstracted information database 40A may store information regarding the position, size, and distance of a predetermined object OBJ within the visual field image of the AR goggles 200.

[4. Configuration example of server device]
Next, the configuration of the server device 100 according to the embodiment will be described using FIG. 5. FIG. 5 is a diagram showing a configuration example of the server device 100 according to the embodiment. As shown in FIG. 5, the server device 100 includes a communication section 110, a storage section 120, and a control section 130.

(Communication Department 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). Further, the communication unit 110 is connected to the network N (see FIG. 2) by wire or wirelessly.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD, an SSD, or an optical disk. As shown in FIG. 5, the storage unit 120 includes a user information database 121, a history information database 122, and a model information database 123.

(User information database 121)
The user information database 121 stores user information regarding the user U. For example, the user information database 121 stores various information such as user U's attributes. FIG. 6 is a diagram showing an example of the user information database 121. In the example shown in FIG. 6, the user information database 121 includes items such as "user ID (identifier),""age,""gender,""home,""worklocation," and "interest."

“User ID” indicates identification information for identifying user U. Note that the "user ID" may be user U's contact information (telephone number, email address, etc.), or may be identification information for identifying user U's terminal device 10.

Moreover, "age" indicates the age of the user U identified by the user ID. Note that the "age" may be information indicating the specific age of the user U (for example, 35 years old, etc.), or may be information indicating the age of the user U (for example, 30s, etc.) . Alternatively, the "age" may be information indicating the date of birth of the user U, or may be information indicating the generation of the user U (for example, born in the 1980s). Furthermore, “gender” indicates the gender of the user U identified by the user ID.

Moreover, "home" indicates the location information of the home of the user U identified by the user ID. In the example shown in FIG. 6, "home" is illustrated as an abstract code such as "LC11," but it may also be latitude and longitude information. Furthermore, for example, "home" may be a region name or address.

Moreover, "work place" indicates the location information of the work place (school in the case of a student) of the user U identified by the user ID. In the example shown in FIG. 6, the "work location" is illustrated as an abstract code such as "LC12," but it may also be latitude and longitude information. Further, for example, the "work location" may be a region name or address.

Moreover, "interest" indicates the interest of the user U identified by the user ID. That is, "interest" indicates an object in which the user U identified by the user ID has a high interest. For example, "interest" may be a search query (keyword) that the user U inputs into a search engine. In the example shown in FIG. 6, one "interest" is shown for each user U, but there may be a plurality of "interests".

For example, in the example shown in FIG. 6, the age of the user U identified by the user ID "U1" is "20s", and the gender is "male". Further, for example, the user U identified by the user ID "U1" indicates that his home is "LC11". Further, for example, the user U identified by the user ID "U1" indicates that the work location is "LC12". Further, for example, the user U identified by the user ID "U1" indicates that he is interested in "sports."

Here, in the example shown in FIG. 6, abstract values such as "U1", "LC11", and "LC12" are used for illustration, but "U1", "LC11", and "LC12" have specific values. It is assumed that information such as character strings and numerical values is stored. Below, abstract values may be illustrated in diagrams related to other information as well.

Note that the user information database 121 is not limited to the above, and may store various information depending on the purpose. For example, the user information database 121 may store various information regarding the terminal device 10 of the user U. In addition, the user information database 121 includes information such as demographic (demographic attributes), psychographic (psychological attributes), geographic (geographical attributes), behavioral (behavioral attributes), etc. of user U. Information regarding attributes may also be stored. For example, the user information database 121 includes name, family composition, place of birth (locality), occupation, position, income, qualifications, type of residence (single-family house, condominium, etc.), presence or absence of a car, commuting/commuting time, commuting/commuting, etc. Memorizes information such as routes, commuter pass sections (stations, lines, etc.), frequently used stations (other than the stations closest to your home or work), lessons (location, time zone, etc.), hobbies, interests, lifestyle, etc. It's okay.

(History information database 122)
The history information database 122 stores various information related to history information (log data) indicating the actions of the user U. FIG. 7 is a diagram showing an example of the history information database 122. In the example shown in FIG. 7, the history information database 122 has items such as "user ID", "location history", "search history", "browsing history", "purchase history", and "posting history".

“User ID” indicates identification information for identifying user U. Further, “position history” indicates a position history that is a history of the user U's position and movement. Further, “search history” indicates a search history that is a history of search queries input by the user U. In addition, “browsing history” indicates a browsing history that is a history of contents that the user U has viewed. Moreover, "purchase history" indicates a purchase history that is a history of purchases by user U. Moreover, "posting history" indicates a posting history that is a history of postings by user U. Note that the "posting history" may include questions regarding user U's belongings.

For example, in the example shown in FIG. 7, the user U identified by the user ID "U1" moves as per "location history #1", searches as per "search history #1", and searches as per "view history #1". This indicates that the content is viewed according to "history #1", a predetermined product, etc. is purchased at a predetermined store etc. according to "purchase history #1", and the content is posted according to "post history #1".

Here, in the example shown in FIG. 7, abstracts such as "U1", "location history #1", "search history #1", "browsing history #1", "purchase history #1" and "posting history #1" are used. Although the figures are illustrated using ``U1'', ``Location history #1'', ``Search history #1'', ``Browsing history #1'', ``Purchase history #1'' and ``Posting history #1'', It is assumed that information such as specific character strings and numerical values is stored.

Note that the history information database 122 is not limited to the above, and may store various information depending on the purpose. For example, the history information database 122 may store the user U's usage history of a predetermined service. Further, the history information database 122 may store the user U's visit history to a physical store, visit history to a facility, or the like. Further, the history information database 122 may store the payment history of user U's payment using the terminal device 10 (electronic payment).

(Model information database 123)
The model information database 123 stores various information regarding an estimation model for estimating abstracted content ABS obtained by abstracting a predetermined object OBJ. FIG. 8 is a diagram showing an example of the model information database 123. In the example shown in FIG. 8, the model information database 123 includes information such as "user attributes,""model,""object,""tolerance,""level of abstraction,""centeredcontent," and "display mode." Has items.

“User attribute” indicates the user attribute of user U (user segment or user persona is also possible). Further, "model" indicates identification information for identifying an estimation model for estimating abstracted content ABS obtained by abstracting a predetermined object OBJ, or the estimation model itself for each user attribute. Further, "object" indicates identification information for identifying a predetermined object OBJ. Further, "tolerance" indicates the user U's tolerance to the predetermined object OBJ. Moreover, "abstraction level" indicates the abstraction level when abstracting the object OBJ. Further, "abstracted content" indicates identification information for identifying abstracted content ABS (abstracted icon, etc.) that is an abstraction of a predetermined object OBJ, or the abstracted content ABS itself (image data, etc.). Furthermore, the “display mode” indicates the display mode of the abstracted content ABS within the visual field image of the AR goggles 200.

For example, in the example shown in FIG. 8, the user U who has the user attribute "Attribute #A" is provided with the estimated model "Model #A", and the estimated model "Model #A" is When a predetermined object OBJ identified by "#A1" is input, "tolerance level #A1", "abstraction level #A1", abstracted content "ABS#A1" and "display mode #A1" are output. shows.

In the example shown in FIG. 8, "attribute #A", "model #A", "OBJ #A1", "tolerance #A1", "abstraction level #A1", "ABS #A1", Although the diagram uses abstract values such as "Aspect #A1", "Attribute #A", "Model #A", "OBJ #A1", "Tolerance #A1", "Abstract level #A1", "ABS #A1” and “Display mode #A1” store information such as specific character strings and numerical values.

Note that the model information database 123 is not limited to the above, and may store various information depending on the purpose. For example, the model information database 123 may store identification information for identifying each user U instead of user attributes of the user U. That is, the estimation model may be prepared not only for each user attribute but also for each individual user. For example, the model information database 123 may store estimated models customized for each user. Furthermore, the model information database 123 may store information regarding the category (classification) of a predetermined object OBJ. Further, the model information database 123 may store information regarding a context when abstracting a predetermined object OBJ.

(Control unit 130)
Returning to FIG. 5, the explanation will be continued. The control unit 130 is a controller, and controls the server device 100 using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). This is realized by executing various programs (corresponding to an example of an information processing program) stored in an internal storage device using a storage area such as a RAM as a work area. In the example shown in FIG. 5, the control unit 130 includes an acquisition unit 131, a determination unit 132, a learning unit 133, and a provision unit 134.

(Acquisition unit 131)
The acquisition unit 131 acquires a search query input by the user U. For example, when the user U inputs a search query into a search engine or the like and performs a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. That is, the acquisition unit 131 acquires, via the communication unit 110, a keyword input by the user U into a search window of a search engine, site, or application.

The acquisition unit 131 also acquires user information regarding the user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information indicating the user U (user ID, etc.), location information of the user U, attribute information of the user U, etc. from the terminal device 10 of the user U. Further, the acquisition unit 131 may acquire identification information indicating the user U, attribute information of the user U, etc. at the time of user U's user registration. The acquisition unit 131 then registers the user information in the user information database 121 of the storage unit 120.

Further, the acquisition unit 131 acquires various types of history information (log data) indicating the behavior of the user U via the communication unit 110. For example, the acquisition unit 131 acquires various types of history information indicating the behavior of the user U from the terminal device 10 of the user U or from various servers based on the user ID or the like. The acquisition unit 131 then registers various types of history information in the history information database 122 of the storage unit 120.

In addition, the acquisition unit 131 acquires the user U (or the user attribute of the user U), the predetermined object OBJ, the tolerance level, the abstraction level, and the abstraction level from the terminal device 10 of the user U via the communication unit 110. A set of content ABS and display mode may be acquired. Note that the acquisition unit 131 may register the acquired information in the model information database 123 of the storage unit 120.

In addition, when performing machine learning in a distributed state without aggregating data by federated learning, the acquisition unit 131 receives learning results ( (difference data or feature amounts) may also be acquired.

(Determination unit 132)
The determining unit 132 determines the user attribute (user segment, user persona, etc.) of the user U from identification information indicating the user U (user ID, etc.). For example, the determination unit 132 determines the user attribute of the user U based on various information (user information, history information, etc.) linked to a user ID common to various services.

(Learning Department 133)
The learning unit 133 performs machine learning using a set of user U's user attribute, predetermined object OBJ, tolerance, abstraction level, abstracted content ABS, and display mode as learning data, and learns user U's user attribute and When a predetermined object OBJ is input, an estimation model is generated/updated to output tolerance, abstraction level, abstracted content ABS, and display mode as estimation results. The learning unit 133 then registers the generated/updated estimation model in the model information database 123 of the storage unit 120.

Note that, in reality, when the learning unit 133 inputs the user attribute of the user U, the predetermined object OBJ, the tolerance level, and the abstraction level, the learning unit 133 creates an estimation model that outputs the abstracted content ABS and the display mode as the estimation result. may be generated/updated. That is, the combination of input data and output data is arbitrary.

Further, the learning unit 133 may integrate the learning results obtained from the terminal device 10 of the user U through federated learning, and generate/update the estimated model.

(Providing unit 134)
The providing unit 134 provides the generated/updated estimation model to the user U who has the target user attributes. That is, the providing unit 134 delivers (distributes) the generated/updated estimation model to the terminal device 10 of the user U according to the user attribute via the communication unit 110.

Further, the providing unit 134 estimates the abstracted content ABS corresponding to the predetermined object OBJ via the communication unit 110 on behalf of the terminal device 10 of the user U using the generated/updated estimation model. , the estimated abstracted content ABS may be provided to the terminal device 10 of the user U.

In addition, when performing machine learning in a distributed state without aggregating data by federated learning, the providing unit 134 transmits not only the generated/updated estimation model itself but also the server device 100 side via the communication unit 110. The terminal device 10 of the user U may be provided with information regarding correction points of the shared model/integrated model.

[5. Processing procedure]
Next, a processing procedure by the terminal device 10 according to the embodiment will be described using FIG. 9. FIG. 9 is a flowchart showing the processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 30 of the terminal device 10.

For example, as shown in FIG. 9, the setting unit 33B of the terminal device 10 abstracts the object OBJ that the user U (user) cannot tolerate, the user U's tolerance level for the object OBJ, and the object OBJ. Settings related to the current level of abstraction are accepted (step S101).

Alternatively, the estimating unit 33C of the terminal device 10 determines the object OBJ that the user U cannot tolerate, the user U The tolerance level and the degree of abstraction when abstracting the object OBJ may be estimated.

Subsequently, the acquisition unit 33A of the terminal device 10 acquires a visual field image photographed by the camera function of the AR goggles 200, which is a glasses-type wearable device, using the short-range wireless communication function or via the communication unit 11 (step S102).

Subsequently, the identifying unit 33D of the terminal device 10 identifies and extracts the object OBJ that is unacceptable to the user U from the visual field image of the AR goggles 200, which is a glasses-type wearable device (step S103).

Next, the generation unit 33E of the terminal device 10 automatically generates the abstracted content ABS from the object OBJ according to the user U's settings regarding the tolerance and abstraction level for the object OBJ (step S104).

At this time, the generation unit 33E may automatically generate the abstracted content ABS from the object OBJ using an estimation model that will be described later.

Subsequently, the conversion unit 33F of the terminal device 10 converts the object OBJ into abstract content ABS acceptable to the user U (step S105).

Subsequently, the display control unit 33G of the terminal device 10 abstracts the position of the object OBJ in the visual field image of the AR goggles 200, which is a glasses-type wearable device, using the short-range wireless communication function or via the communication unit 11. The content ABS is displayed in a superimposed manner (step S106).

Subsequently, the transmitting unit 31 of the terminal device 10 transmits information (learning data) via the communication unit 11 regarding the set of the user U, the predetermined object OBJ, the tolerance level, the abstraction level, the abstracted content ABS, and the display mode. ) is transmitted (provided) to the server device 100 (step S107).

Subsequently, when the reception unit 32 of the terminal device 10 receives the predetermined object OBJ, tolerance level, and abstraction level as a response from the server device 100 via the communication unit 11, it receives the abstracted content ABS as an estimation result. and a display mode are received (step S108).

Subsequently, the processing unit 33 of the terminal device 10 stores the received estimation model in the storage unit 40 so that the generation unit 33E can automatically generate abstracted content ABS from the predetermined object OBJ using the estimation model. (Step S109).

[6. Modified example]
The terminal device 10 and server device 100 described above may be implemented in various different forms other than the above embodiments. Therefore, a modification of the embodiment will be described below.

In the embodiment described above, a part or all of the processing that is being executed by the terminal device 10 of the user U may actually be executed by the AR goggles 200. For example, the AR goggles 200 may complete the process stand-alone (AR goggles 200 alone). In this case, it is assumed that the AR goggles 200 have the functions of the terminal device 10 in the above embodiment.

Further, in the above embodiment, part or all of the processing that is being executed by the terminal device 10 of the user U may actually be executed by the server device 100. For example, data may be collected and processed from the terminal device 10 of each user U to the server device 100. In this case, it is assumed that the server device 100 has the functions of the terminal device 10 in the above embodiment. Furthermore, in the embodiment described above, since the server device 100 cooperates with the terminal device 10, from the user U's perspective, there is no difference even if the server device 100 executes the processing of the terminal device 10.

Further, in the embodiment, a part or all of the processing executed by the server device 100 may actually be executed by the terminal device 10. For example, the terminal device 10 may generate the estimation model through on-device learning. That is, the processing may be completed on the terminal device 10 without communication between the terminal device 10 and the server device 100. In this case, it is assumed that the terminal device 10 has the functions of the server device 100 in the above embodiment. Further, in the embodiment described above, since the terminal device 10 cooperates with the server device 100, it appears to the user U that the terminal device 10 is also executing the processing of the server device 100. That is, from another point of view, it can be said that the terminal device 10 includes the server device 100.

Furthermore, in the embodiment, the terminal device 10 of the user U may be able to set, as a predetermined object, something that the user does not want other users to see. For example, the predetermined object may be something that you do not want to show to children, or it may be confidential information or confidential information that you do not want to show to anyone other than your company. In this case, the user U uses the terminal device 10 to set a predetermined object that the user U does not want to show to users other than the users he or she has authorized (authorized users). The terminal device 10 provides and registers the settings of the authorized user and the predetermined object to the server device 100. The server device 100 reflects the settings of the predetermined object on the terminal devices 10 of users other than the authorized users. The settings of this predetermined object may be changed only by the user U (and authorized users) who have set the object, and may not be changed by other users. For example, when an outsider wears the AR goggles 200 to take a tour of the company or a factory, objects that are confidential to the company or that are subject to confidentiality will be treated according to the tolerance and abstraction level set by the user U. It may be displayed in an abstract manner.

[7. effect〕
As described above, the information processing device (terminal device 10, server device 100, or AR goggles 200) according to the present application includes an acquisition unit 33A that acquires a visual field image of a glasses-type wearable device, and a user U from the visual field image. a specifying unit 33D that identifies an object OBJ that is unacceptable; a converting unit 33F that converts the object OBJ into abstracted content ABS that is acceptable to the user U; The display controller 33G includes a display control unit 33G that superimposes and displays the abstracted content ABS at a position.

The information processing device according to the present application further includes a setting unit 33B that accepts settings of an object OBJ that is unacceptable to the user U (user), and a storage unit that stores a set of an object OBJ and an abstracted content ABS.

The information processing device according to the present application further includes an estimation unit 33C that estimates an object OBJ that is unacceptable to the user U from the user U's attribute information. The conversion unit 33F converts the object OBJ into abstracted content ABS acceptable to the user U.

The information processing device according to the present application further includes a generation unit 33E that automatically generates abstracted content ABS from object OBJ. The conversion unit 33F converts the object OBJ into abstracted content ABS.

The information processing device according to the present application further includes a setting unit 33B that receives settings regarding user U's tolerance to object OBJ. The conversion unit 33F converts the object OBJ into abstracted content ABS according to the tolerance level.

The information processing device according to the present application further includes an estimation unit 33C that estimates the tolerance of the user U to the object OBJ from the user U's attribute information. The conversion unit 33F converts the object OBJ into abstracted content ABS according to the tolerance level.

The information processing device according to the present application further includes a setting unit 33B that receives settings regarding the abstraction level when abstracting the object OBJ. The conversion unit 33F converts the object OBJ into abstracted content ABS according to the level of abstraction.

The information processing device according to the present application further includes an estimation unit 33C that estimates the degree of abstraction when abstracting the object OBJ from the attribute information of the user U. The conversion unit 33F converts the object OBJ into abstracted content ABS according to the level of abstraction.

The setting unit 33B inquires of the user U about the degree of preference for the object OBJ, receives a response from the user U to the inquiry about the degree of preference for the object OBJ, and further provides the user U with the converted abstracted content ABS. Inquire about the degree of preference for the abstracted content ABS after conversion, receive answers from the user U to the inquiry about the degree of preference for the converted abstract content ABS, and change the degree of abstraction when abstracting the object OBJ according to each answer. . The conversion unit 33F converts the object OBJ into abstracted content ABS according to the level of abstraction.

By using any one or a combination of the above-described processes, the information processing apparatus according to the present application can abstract what the user does not want to see in the visual field image of the glasses-type wearable device.

[8. Hardware configuration]
Further, the terminal device 10 and the server device 100 according to the embodiments described above are realized by, for example, a computer 1000 having a configuration as shown in FIG. 10. The following will explain the terminal device 10 as an example. FIG. 10 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are connected to a bus. 1090.

The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and performs various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used by the calculation device 1030 for various calculations. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, and includes a ROM (Read Only Memory), an HDD (Hard Disk Drive), and an SSD (Solid Disk Drive). This is realized using flash memory, etc. The secondary storage device 1050 may be a built-in storage or an external storage. Further, the secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. Further, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface) and HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, button, scanner, etc., and is realized by, for example, a USB or the like.

Further, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, output device 1010 and input device 1020 may be wireless devices.

Moreover, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends it to computing device 1030, and also sends data generated by computing device 1030 to other devices via network N.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the terminal device 10, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 30 by executing a program loaded onto the primary storage device 1040. Further, the arithmetic device 1030 of the computer 1000 may load a program obtained from another device via the network I/F 1080 onto the primary storage device 1040, and execute the loaded program. Furthermore, the arithmetic unit 1030 of the computer 1000 may cooperate with other devices via the network I/F 1080, and may call and use program functions, data, etc. from other programs of other devices.

[9. others〕
Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-mentioned components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that are in a so-called equivalent range. Furthermore, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the constituent elements can be made without departing from the gist of the embodiments described above.

Further, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

For example, the server device 100 described above may be realized by a plurality of server computers, and depending on the function, it may be realized by calling an external platform etc. using an API (Application Programming Interface), network computing, etc. Can be changed flexibly.

Furthermore, the above-described embodiments and modifications can be combined as appropriate within a range that does not conflict with the processing contents.

Further, the above-mentioned "section, module, unit" can be read as "means", "circuit", etc. For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

1 Information processing system 10 Terminal device 33 Processing unit 33A Acquisition unit 33B Setting unit 33C Estimation unit 33D Specification unit 33E Generation unit 33F Conversion unit 33G Display control unit 40 Storage unit 40A Abstracted information database 100 Server device 110 Communication unit 120 Storage unit 121 User information database 122 History information database 123 Model information database 130 Control unit 131 Acquisition unit 132 Judgment unit 133 Learning unit 134 Providing unit

Claims (11)

an acquisition unit that acquires a visual field image of the glasses-type wearable device;
a specifying unit that specifies an object that is unacceptable to the user from the visual field image;
a conversion unit that converts the object into abstracted content acceptable to the user;
An information processing apparatus comprising: a display control unit that displays the abstracted content in a superimposed manner at the position of the object in the field of view image of the glasses-type wearable device. a setting unit that accepts settings for objects that are unacceptable to the user;
The information processing apparatus according to claim 1, further comprising: a storage unit that stores a set of the object and the abstracted content. further comprising an estimating unit that estimates an object that is unacceptable to the user from the user's attribute information,
The information processing apparatus according to claim 1, wherein the conversion unit converts the object into the abstracted content that is acceptable to the user. further comprising a generation unit that automatically generates the abstracted content from the object,
The information processing device according to claim 1, wherein the conversion unit converts the object into the abstracted content. further comprising a setting unit that accepts settings regarding the user's tolerance to the object,
The information processing apparatus according to claim 1, wherein the conversion unit converts the object into the abstracted content according to the tolerance. further comprising an estimating unit that estimates the user's tolerance to the object from the user's attribute information,
The information processing apparatus according to claim 1, wherein the conversion unit converts the object into the abstracted content according to the tolerance. further comprising a setting unit that accepts settings regarding the abstraction level when abstracting the object,
The information processing device according to claim 1, wherein the conversion unit converts the object into the abstracted content according to the abstraction level. The setting section includes:
Inquiring the user about the degree of preference for the object, receiving a response from the user to the inquiry about the degree of preference for the object,
further, inquiring the user about the degree of preference of the abstracted content after conversion, and receiving a response from the user to the inquiry about the degree of preference of the abstracted content after conversion;
Depending on each answer, the degree of abstraction when abstracting the object is changed,
The information processing device according to claim 7, wherein the conversion unit converts the object into the abstracted content according to the abstraction level. further comprising an estimation unit that estimates a degree of abstraction when abstracting the object from the user's attribute information,
The information processing device according to claim 1, wherein the conversion unit converts the object into the abstracted content according to the abstraction level. An information processing method executed by an information processing device, the method comprising:
an acquisition step of acquiring a visual field image of the glasses-type wearable device;
an identification step of identifying an object that is unacceptable to the user from the visual field image;
a conversion step of converting the object into abstracted content acceptable to the user;
An information processing method comprising: a display control step of superimposing and displaying the abstracted content at the position of the object in the field of view image of the glasses-type wearable device. an acquisition means for acquiring a visual field image of the glasses-type wearable device;
identification means for identifying an object unacceptable to the user from the visual field image;
converting means for converting the object into abstracted content acceptable to the user;
and display control means for superimposing and displaying the abstracted content at the position of the object in the field of view image of the glasses-type wearable device.

Images