WO2022124419A1

WO2022124419A1 - Information processing apparatus, information processing method, and information processing system

Info

Publication number: WO2022124419A1
Application number: PCT/JP2021/045713
Authority: WO
Inventors: 聡黒田
Original assignee: 株式会社情報システムエンジニアリング
Priority date: 2020-12-11
Filing date: 2021-12-10
Publication date: 2022-06-16
Also published as: JPWO2022124419A1

Abstract

[Problem] To provide an information processing apparatus, an information processing method, and an information processing system which enable, when needed by an operator, provision of necessary amount of information to the operator and a co-operator, without requiring large-scale information reconstruction. [Solution] The present invention provides an information processing apparatus which is for outputting work information relating to work to be performed by an operator, and which is provided with: an image acquisition unit that acquires original images which include subject persons including an operator and an operation-receiving person and include a plurality of operation items to be handled by the operator; an image division unit that divides the original images into a subject person image in which the subject persons are captured of and a plurality of operation item images in which the operation items are captured of; a scene deduction unit that deduces a scene by using a first trained model; a chunk deduction unit that deduces a chunk by using one of a plurality of second trained models; an output unit that outputs the chunk; a recommendation image output unit that searches for a recommending operation item image and outputs a recommendation image; and a display unit that allocates the outputted chunk and the outputted recommendation image to an object model display area and displays the same. The chunk deduction unit selects one of the plurality of the second trained models by using a model ID associated with a scene ID on a one-by-one basis. A meta ID for a chunk uniquely indicates a meta value for the chunk, which is information relating to the property of an operation item.

Description

Information processing equipment, information processing methods and information processing systems

The present invention relates to an information processing apparatus, an information processing method and an information processing system.

For example, in the work support system of Patent Document 1, a rule describing the judgment conditions of the work target or the work situation is generated based on the manual describing the work procedure, contents, points to be noted or other matters, and the worker. Recognizes the work target and work status based on the sensor information from the device worn by the user, and outputs work support information based on the recognition result of the generated rule and recognition means.

Japanese Unexamined Patent Publication No. 2019-109844

However, in the conventional method as described in Patent Document 1, the information stored as a document such as a manual can only be searched for each document. For example, if you want to search a document paragraph by paragraph, you need to reconstruct the document into structured information. Reconstruction of all documents to be searched is often not realistic in terms of cost-effectiveness, and information on a document-by-document basis often browses unnecessary information, so that the viewer of the document can quickly browse. There is a problem that it may not be possible to deal with it.

One aspect of the embodiment of the present invention is information processing that presents the required amount of information to the responder and the collaborator when the responder needs it, without reconstructing the information on a large scale. It is an object of the present invention to provide an apparatus, an information processing method and an information processing system.

An information processing device that outputs work information that is information about work performed by a responder, and is a plurality of objects corresponding to the responder and a target person including at least one of the respondents to which the responder corresponds. An image acquisition unit that acquires an original image that is an image including the corresponding object, a target image obtained by dividing the original image and captured by the target person, and a plurality of corresponding object images obtained by capturing each corresponding object. Using the first trained model in which the relationship between the image division unit that divides into and the image of the target person, the scene ID that uniquely indicates the scene that the correspondent performs, and the scene ID are stored. , A scene estimation unit that estimates a scene, a plurality of object images, a chunk ID that uniquely indicates a chunk that is information that divides or suggests work information, and one or a plurality of chunks that are associated with one-to-one. A chunk estimator that estimates chunks and a chunk output unit that outputs chunks using one of a plurality of second trained models in which the association between the meta IDs is stored. , One of a plurality of second trained models is recommended as a search key using a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID in a one-to-one manner. A recommendation image output unit that searches for an image and outputs a recommended image that is an image of a corresponding object that is not captured in the original image but is presumed to be necessary, and chunks and chunks output by the chunk output unit. Each of the recommended images output by the recommended image output unit is provided with a display unit for allocating and displaying each of the recommended images to each surface of the object model having a plurality of display areas, and the chunk estimation unit is a plurality of second trained models. One of them is selected using the scene ID and the model ID associated with one-to-one, and the chunk meta ID uniquely indicates the chunk meta value which is information on the property of the corresponding object. Provides a processing device.

It is an information processing method performed by an information processing device that outputs work information that is information about the work performed by the responder, and the responder and the respondent include at least one of the respondents and a plurality of respondents. The first step of acquiring the original image which is an image including an object, the subject image obtained by dividing the original image and the subject imaged, and the plurality of object images captured by each corresponding object. Using the first trained model, which stores the association between the second step of division, the subject image, and the scene ID that uniquely indicates the scene that the correspondent is doing, A third step of estimating the scene, a plurality of object images, a chunk ID uniquely indicating a chunk that is information that divides or suggests work information, and one or more chunks associated with one-to-one. A fourth step of estimating chunks and a fifth of outputting chunks using one of a plurality of second trained models in which the association between the meta IDs is stored. A step and one of a plurality of second trained models are recommended as a search key using a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID in a one-to-one manner. The sixth step of searching the corresponding object image and outputting the recommended image which is the image of the corresponding object which is not captured in the original image but is presumed to be necessary, and the chunk and the recommended image. Each is provided with a seventh step of assigning and displaying each side of an object model having a plurality of display areas, and one of the plurality of second trained models has a one-to-one correspondence with the scene ID. The chunk meta ID, which is selected using the attached model ID, provides an information processing method that uniquely indicates the chunk meta value which is information about the property of the corresponding object.

An information processing system that outputs work information that is information about work performed by a responder, and identifies the responder, a target person including at least one of the respondents to which the responder corresponds, and the target person. An image acquisition means for acquiring an original image which is an image including at least one of the target person identification information and a plurality of corresponding objects to which the corresponding person corresponds, and the target person using the original image. An image dividing means for dividing an image into a plurality of images of the corresponding objects captured by each of the corresponding objects, an image of the target person, and a scene ID uniquely indicating a scene in which the corresponding object is performed. Using the first trained model in which the relationships between the two are stored, the scene estimation means for estimating the scene, the plurality of objects to be imaged, and the information obtained by dividing or suggesting the work information. One of a plurality of second trained models in which the association between a chunk ID uniquely indicating a certain chunk and one or more chunk meta IDs associated with one-to-one is stored. One of the chunk estimation means for estimating the chunk, the chunk output means for outputting the chunk, and the plurality of second trained models has a one-to-one correspondence with the scene ID. Using the attached model ID and a combination of one or more chunk meta IDs as a search key, the recommended recommended object image and the shared shared object information are searched, and the original image is captured. Includes images of the subject that are not, but are presumed to be necessary, the subject and information about the subject's work that is presumed to require information sharing in the subject's work. The recommendation image output means for outputting the recommendation image, the chunk output by the chunk output means, and the recommendation image output by the recommendation image output means are assigned to the display area of the object model having a plurality of display areas. The chunk estimation means is selected by using the model ID, and the chunk meta ID uniquely indicates a chunk meta value which is information about the property of the corresponding object. Provides an information processing system.

According to one aspect of the embodiment of the present invention, an information processing device that presents the required amount of information to the responder when the responder needs it, without reconstructing the information on a large scale. , Information processing method and information processing system can be realized.

FIG. 1 is a block diagram showing a configuration of an information processing apparatus at a utilization stage according to the present embodiment. FIG. 2 is a block diagram showing a configuration of an information processing apparatus in the learning stage according to the present embodiment. FIG. 3 is a diagram showing an original image, a subject image, and a plurality of objects to be imaged according to the present embodiment. FIG. 4 is a diagram showing a tree structure which is a relationship between a subject image and a plurality of objects to be imaged according to the present embodiment. FIG. 5 is a diagram showing a first trained model and a second trained model according to the present embodiment. FIG. 6 is a diagram showing information stored in the auxiliary storage device according to the present embodiment. FIG. 7 is a sequence diagram for explaining the scene estimation function, chunk estimation function, and chunk output function according to the present embodiment. FIG. 8 is a sequence diagram provided for explaining the first trained model generation function and the second trained model generation function according to the present embodiment. FIG. 9 is a flowchart showing a processing procedure of information processing in the usage stage according to the present embodiment. FIG. 10 is a flowchart showing a processing procedure of information processing in the display stage according to the present embodiment. FIG. 11 is a flowchart showing a processing procedure of information processing in the learning stage according to the present embodiment. FIG. 12A is a schematic diagram showing an example of the operation of the information processing system according to the second embodiment, and FIG. 12B is an original image, a subject image, and a plurality of objects to be supported according to the second embodiment. It is a figure which shows the image. FIG. 13 is a diagram showing information stored in the auxiliary storage device according to the present embodiment. FIG. 14 is a diagram showing information stored in the auxiliary storage device according to the present embodiment. FIG. 15 is a diagram showing information stored in the auxiliary storage device according to the present embodiment. FIG. 16 is a diagram showing information stored in the auxiliary storage device according to the present embodiment. 17 (a) to 17 (h) are diagrams showing a display pattern in a subject according to the second embodiment. 18 (a) and 18 (b) are schematic views showing an example of a display of a user terminal according to the second embodiment.

Hereinafter, one aspect of the embodiment of the present invention will be described in detail with reference to the drawings. For example, at a university window or a pharmacy window, including a target person who is a student, a guardian of a student, a patient, etc., and a person who has a different position or role, such as a responder who is a worker who handles the target person. The information on the object to be referred to by the responder will be explained. The corresponding object is, for example, a document in the case of a university window and a drug in the case of a pharmacy window.

In addition to, for example, a university window or a pharmacy window, in a work place where multiple workers such as factories collaborate to perform one or more tasks, the workers are mainly the workers to perform the tasks. , Other workers for the work, customers who receive the work, etc. may be the target people, including those who have different positions and roles, and multiple people who share the work at the site. In this case, the object is, for example, a device, a product, or a part installed at a work site or a place.

(The present embodiment: the first embodiment)
The information processing system 100 in this embodiment has an information processing device 1 as shown in FIG. 1, for example. First, the information processing apparatus 1 in the usage stage will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of an information processing apparatus 1 in a utilization stage according to the present embodiment. The information processing device 1 includes a central processing unit 2, a main storage device 3, and an auxiliary storage device 11.

The central processing unit 2 is, for example, a CPU (Central Processing Unit), and executes processing by calling a program stored in the main storage device 3. The main storage device 3 is, for example, a RAM (RandomAccessMemory), which is an image acquisition unit 4, an image division unit 5, a scene estimation unit 6, a chunk estimation unit 7, a chunk output unit 8, and a first trained model, which will be described later. A program such as a generation unit 9, a second learned model generation unit 10, a recommendation image output unit 13, and an object model identification unit 15 is stored.

A program including an image acquisition unit 4, an image division unit 5, a scene estimation unit 6, a chunk estimation unit 7, a chunk output unit 8, and a recommendation image output unit 13 may be called a control unit 15, and the first trained model generation may be performed. A program including the unit 9 and the second trained model generation unit 10 may be referred to as a trained model generation unit 16.

The auxiliary storage device 11 is, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive), such as a first trained model DB 1, a first learning model DB 1', or a second trained model DB 2, which will be described later. Databases such as the second learning model DB2', scene table TB1, model table TB2, content table TB3, scene content table TB4, content chunk table TB5, chunk metatable TB6, chunk table TB7, chunk metatable TB8, etc. Stores tables such as the recommendation table TB9, the object table TB10, the object allocation table TB11, the annotation table TB12, the attention table TB13, the camera table TB14, and the roll table TB15.

As shown in FIG. 1, the information processing apparatus 1 that outputs work information that is information about the work performed by the corresponding person has an image acquisition unit 4, an image division unit 5, a scene estimation unit 6, and chunk estimation at the usage stage. A unit 7 and a chunk output unit 8 that outputs chunks that are information that divides or suggests work information are provided. Here, the work information may be referred to as content, and the content ID uniquely indicates the work information.

The image acquisition unit 4 is an original image 20 (FIG. 3) which is an image including a corresponding person 21 (FIG. 3) corresponding to the corresponding person and a plurality of corresponding objects 22 to 25 (FIG. 3) corresponding to the corresponding person. Is acquired from a user terminal 12 such as a personal computer equipped with a camera. The image segmentation unit 5 divides the original image 20 into an image of the person to be imaged by the person to be dealt with 21, an object identification information 61 for identifying the object (such as a person to be correspondent or a person to be correspondent), and each subject. The corresponding objects 22 to 25 are divided into a plurality of imaged objects 40 to 43.

The scene estimation unit 6 estimates the scene that is the situation performed by the responder. Specifically, the scene estimation unit 6 acquires, for example, the corresponding person image 30 (35), for example, the corresponding person image, the target person identification information 61, and the like as the target person image. The scene estimation unit 6 uses, for example, the first trained model DB1 in which the association between the corresponding person image 30 (35) and the scene ID uniquely indicating the scene is stored. And estimate the scene. In addition to the corresponding person image 30 (35), the scene estimation unit 6 uses, for example, an image of the corresponding person (target person image), identification information for identifying the corresponding person and the corresponding person (target person identification information), and a scene. The scene may be estimated using the first trained model DB1 in which the association between the uniquely shown scene ID and the scene ID is stored.

For example, when the target person identification information is the corresponding person identification information, the scene estimation unit 6 has a relationship between the corresponding person identification information and the scene ID uniquely indicating the scene in which the corresponding person performs the situation. The first trained model in which is stored may be used to further estimate the scene.

For example, when the person to be corresponded does not exist (when the target person identification information is the person identification information), the scene estimation unit 6 uniquely indicates the person identification information and the scene which is the situation performed by the person. A first trained model in which the association between and is stored may be used to further estimate the scene.

The scene estimation unit 6 acquires a scene name using the scene ID as a search key from the scene table TB1 which is a table in which the scene ID and the scene name which is the name of the scene are linked one-to-one, and the user terminal 12 is used. Send. The user terminal 12 presents the scene name received from the scene estimation unit 6 to the target person. The presentation of the scene name is displayed, for example, on one side of the object model previously assigned by the target person by the display unit 17 described later.

The chunk estimation unit 7 acquires the corresponding object images 40 to 43, which are images of the corresponding object 22 (23 to 25) related to the work, and uniquely identifies the chunk with the corresponding object images 40 (41 to 43). Using one of a plurality of second trained model DB2s in which the association between the indicated chunk ID and one or more chunk meta-IDs associated one-to-one is stored. , Estimate chunks.

The chunk estimation unit 7 selects one of a plurality of second trained models in which the association between the plurality of object images, the chunk ID, and the plurality of chunk meta IDs is stored. For example, when the target person information is the corresponding person identification information, the chunk associated with the corresponding person identification information may be further estimated.

Further, the chunk estimation unit 7 is one of a plurality of second trained models in which the association between the plurality of object images, the chunk ID, and the plurality of chunk meta IDs is stored. For example, when the target person information is the corresponding person identification information, the chunk associated with the corresponding person identification information may be further estimated.

The chunk estimation unit 7 selects one of the plurality of second trained model DB2s by using the model ID associated with the scene ID on a one-to-one basis. Further, the chunk meta ID uniquely indicates a chunk meta value which is information regarding the properties of the corresponding objects 22 to 25.

The chunk estimation unit 7 acquires the model ID from the model table TB2, which is a table in which the model ID and the scene ID are linked one-to-one, using the scene ID as a search key. Further, the chunk estimation unit 7 acquires the chunk ID from the chunk meta table TB6, which is a table in which the chunk ID and the chunk meta ID are linked one-to-one or one-to-many, using the chunk meta ID as a search key. ..

Further, the chunk estimation unit 7 acquires a chunk summary showing the outline of the chunk from the chunk table TB7 using the chunk ID as a search key, and transmits the chunk summary to the user terminal 12. The user terminal 12 presents the chunk summary received from the chunk estimation unit 7 to the target person. The presentation of the chunk summary is displayed, for example, on one side of the object model previously assigned by the target person by the display unit 14 described later.

Further, the chunk estimation unit 7 acquires chunks from the chunk table TB7 using the chunk ID as a search key, and transmits the chunks to the user terminal 12. The user terminal 12 presents the chunk received from the chunk estimation unit 7 to the target person. The chunk presentation is displayed, for example, on one side of the object model previously assigned by the target person by the display unit 14 described later.

The chunk table TB7 is a table in which chunks, chunk summaries, and hash values are associated with a chunk ID on a one-to-one basis. The hash value is used, for example, to confirm whether or not the chunk has been changed.

The image acquisition unit 4 acquires the target person identification information for identifying the target person. The target person identification information is, for example, a face image that identifies the responder and the respondent, and is, for example, a bar code of an ID card such as a photo certificate, a two-dimensional code, or the like. It may be imaged by a camera or the like. The information processing apparatus 1 identifies the captured target person identification information and confirms that the responder is a legitimate target person (correspondent) for the work. The target person identification information may, for example, identify a plurality of target persons, and may further identify a remote co-owner who shares information.

Here, the information stored in the auxiliary storage device 11 according to the present embodiment will be described with reference to FIGS. 13 to 16.

FIG. 13 shows the object model table TB10. The object model table TB10 displays, for example, an object model ID for identifying the object model, operation information indicating the type of operation for the object model, a basic size in which the object model is displayed, and a plurality of object models 6 as information for identifying the object model. Possible additional number, display coordinates indicating the position where the object model is displayed, estimated scene ID and chunk ID, area ID (affiliation / department) that identifies the affiliation, department, location, etc. to perform work, target person (correspondence) Persons, respondents, etc.) and role IDs that identify the skills, attributes, roles, qualifications, etc. of the responders are stored in association with each other.

Next, FIG. 14 shows the object allocation table TB11. The object allocation table TB11 is, for example, linked to the object model ID, display area information regarding the number of display areas of the object model, display area ID to which the recommended image to be displayed and reference information are linked, scene ID, chunk ID. Etc. are stored in association with each.

Next, FIG. 15 shows the annotation table TB12 and the attention table TB13. In the annotation table TB12, for example, a video ID for identifying a video related to work, a camera ID for identifying a camera that shot, a scene ID for identifying a work scene, a shooting time indicating the time when the video was shot, and a video were shot. The shooting time, image quality information, image quality and viewpoint coordinates of the shot image, meta ID, attention ID, etc. are stored in association with each other.

The attention table TB13 is, for example, an attention ID that identifies the priority of chunks to be displayed, attention type information indicating the display of attention information, a scene ID, attention information that occupies the content, attention information data (content), and higher reference information. , High-ranking attention ID indicating the presence or absence, etc. are stored in association with each.

Next, FIG. 16 shows the camera table TB14 and the roll table TB15. The camera table TB14 can be operated, for example, a camera ID that identifies the camera to be photographed, an area ID in which the camera is installed, model information indicating camera specifications, operations, roles, etc., line-of-sight information, switching information, external connection information, and operation. Person ID, role ID, etc. are stored in association with each other. In addition, the role table TB15 has a role ID, an employee ID, a name, a qualification ID, a department ID, an area ID, a related role ID, etc. that identify a target person (correspondent, respondent, other responder, co-owner, etc.). , Correspond to each and stored.

The information processing device 1 may further include a recommendation image output unit 13, and the auxiliary storage device 11 may further include a recommendation table TB9. The recommendation image output unit 13 searches for a recommended object image using the recommendation table TB9 using a combination of the model ID and one or a plurality of chunk meta IDs as a search key.

The recommendation image output unit 13 outputs the searched recommended object image to the user terminal 12. The recommended object to be image refers to an image of the object to which the original image 20 is not captured, but is presumed to be necessary. The recommendation table TB9 is a table in which the combination of the model ID and the chunk meta ID and the recommended object image are linked one-to-one-to-one.

The recommendation image output unit 13 searches for one of the plurality of second trained models as a search key for a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID on a one-to-one basis. As a method, the shared person who shares information in the work and the shared information shared with the shared person may be further searched, and the recommended information associated with the shared person and the shared information may be output.

Further, the recommendation image output unit is, for example, a collaborator who collaborates with the target person (correspondent), a trainer who is a leader of the target person (correspondent), or at least an inspector who monitors the target person (correspondent). A person identified in any position may be searched as a shared person, and the recommended information associated with the shared person and the shared information may be output.

The display unit 14 allocates each of the chunks and the recommended image output by the chunk output unit 13 to each surface of the object model having a plurality of display areas, and assigns the assigned object model. It is displayed on the user terminal 12 via the user terminal 12.

The display unit 14 further includes an object model identification unit 15. The object model specifying unit 15 identifies the object model that displays the recommended image and the recommended information output by the recommended image output unit 13, and associates the scene and chunk with the object model ID that uniquely indicates the object model. , Identify the object model.

The display unit 14 displays the recommendation image output by the recommendation image output unit 13 and the recommendation information in any of the display areas of the plurality of display areas included in the object model specified by the object model identification unit 15. , Assign as a state that can be shared with the shared person.

The object model specifying unit 15 uniquely displays an object model that displays reference information including at least a scene or chunk based on at least one of a scene estimated by the scene estimation unit 6 and a chunk estimated by the chunk estimation unit 7. The object model is specified by associating with the indicated object model ID.

The object model specifying unit 15 refers to the object model table TB10 acquired by, for example, the image acquisition unit 4 and stored in the auxiliary storage device 11, and displays reference information narrowed down to the corresponding person when the corresponding person needs it. Identify a suitable object model 6 to do.

Specifically, the object model specifying unit 15 refers to the object model table TB10 shown in FIG. 13, and based on, for example, an estimated scene ID, chunk ID, etc., based on various information such as a work area and a position of a corresponding person. Identify the object model that can be displayed under that condition.

The object model specifying unit 15 is aware of the information narrowed down to the target person and the corresponding person when the corresponding person is required by, for example, the basic size, display coordinates, area ID, etc. of the object model identified by the object model ID. Identify an object model that can present information to the responder that displays no useful information.

For example, the object model specifying unit 15 identifies an object model, for example, a qualification ID for specifying skill information of a correspondent, display coordinates for specifying spatial information for performing work, a basic size for specifying feature information of an object, or work. It may be specified based on at least one of the work level information of.

The object model specifying unit 15 refers to the object model table based on the estimated scene ID, chunk ID, etc., and for example, from various data such as the shape, the basic size, and the number of additional objects associated with the object model ID, at least 2 It is also possible to specify an object model having a shape having a display area 8 equal to or larger than a surface, and capable of displaying a plurality of the same or different object models 6.

The object model specifying unit 15 refers to the object model table TB10 based on the estimated scene ID, chunk ID, etc., and for example, by various operation information for the object model 6, rotation display, enlarged display, based on the working state, You may want to specify an object model that can be displayed in at least one of reduced display, protruding display, vibration display, state display, discoloration display, and shading display.

The object model specifying unit 15 may be specified based on various information such as the positional relationship between the object and the corresponding person, the dominant hand, the language used, and the like, and the displayed position may be determined. The object model specifying unit 15 identifies an object model having a display area that can be displayed based on, for example, the type of chunk estimated by the chunk estimation unit 7, the number of characters in the chunk, and the like, and the chunks are displayed in each display area of the object model. You may make an assignment.

For example, when the object model displayed in advance by the corresponding person is customized and registered, the object model specifying unit 15 may preferentially display the customized object model.

The object model may be rotated or moved by grasping any of the left and right or upper and lower sides of the object model, for example, via the user terminal 12 worn by the corresponding person.

For example, when it is necessary to reliably present information such as emergency information for various assigned information, the object model specifying unit 15 projects the specified object model toward the corresponding person and displays it. May be good.

The display area of the object model specified by the object model specifying unit 15 includes at least a scene or chunk based on at least one of the scene estimated by the scene estimation unit 6 and the chunk estimated by the chunk estimation unit 7. Assign reference information.

Specifically, the object model specifying unit 15 refers to the object allocation table TB11 shown in FIG. 14, for example, the estimated scene ID, the displayable information indicated by the chuck ID, the work area, the position of the correspondent, and the chunk data. Based on various information such as the amount, the object model specified by the object model specifying unit 15 is assigned to the display area.

For example, when the object model specifying unit 15 allocates the estimated chunks, the object model specifying unit 15 may refer to the attention table TB13 shown in FIG. 15 in addition to the object allocation table and allocate the chunks associated with the scene ID. good. In this case, for example, the attention ID of the attention table TB13 is referred to, and the presence or absence of the attention information set in association with the image of the object to be matched is determined.

Based on the attention IDs of the annotation table TB12 and the attention table TB13 shown in FIG. 15, the object model identification unit 15 has reference information (for example, a display having priority over the estimated chunk, or a display attached to the chunk). (Attention information, attention information data, etc.) are assigned. As a result, when the responder needs it, the information that attracts more attention than the information that displays the information narrowed down to the responder and the useful information that the responder is not aware of, or the information at the higher level is attached or prioritized. It can be presented to the target person as a display by an object model such as that, and it is possible to improve the effectiveness of information provision and the usefulness of information.

The object model specifying unit 15 refers to the annotation table TB12 based on the estimated scene ID, chunk ID, etc., and for example, all or part of various videos and data associated with each of the attention IDs are the videos. May be assigned based on the time, length, and viewpoint at which the image was taken.

In addition to the original image, the object image, and the target person identification information acquired by the image acquisition unit 4, for example, the position information of the room or place where the person in charge works, the environmental information such as the ambient temperature and humidity, the work of the target person, and the work of the target person. The display of the display area in the object model may be assigned based on various information related to the movement, the biological information of the target person, and the like.

The object model specifying unit 15 has an object model, each display area of the object model, and a display area of the object model so that the virtual reality displayed on the transmissive display of the user terminal 12 is superimposed and displayed to the corresponding person. Set the recommended image to be assigned to the display area.

The object model specifying unit 15 may acquire evaluation target information including at least position information which is information on the position where a corresponding person is present and work-related information related to work. The work-related information is, for example, information around the object to be addressed, the dominant arm that the responder (target person) works on, and the like, based on these, for example, "the work position of the responder is to the right of the object to be addressed", "The distance between the correspondent and the object to be dealt with is 3 meters", "there is no arrangement around the object to be correspondent", "the dominant arm of the correspondent is to the right", and the like may be displayed.

When the object model specifying unit 15 determines that, for example, "the space on the left side of the object to be corresponded is empty", "the correspondent is right-handed", and "the estimated chunk amount is 2 screens", the object model identification unit 15 is to be dealt with. "Cube_02 (cube)", "allocated to 6 areas", "object model size (W: D: H)" and "display coordinates on the left side of the corresponding object (XX: YY: ZZ)" that can be displayed in the space on the left side of the object. ) ”,“ Chunk (1B827-01.txt_0) ”and the like are set in the display area.

The object model specifying unit 15 is an object model specified based on, for example, a scene as a work place, a positional relationship between a correspondent and a corresponding object, chunk data or information amount as reference information, a size of a recommended image, and the like. In the case of a plurality of cubes, for example, a plurality of cubes may be positioned vertically or horizontally and displayed.

In addition to the estimated chunks, the object model specifying unit 15 may assign and display a working state or the like on the upper surface of the object model, for example, as a "face mark" indicating emotions and emotions of the face. The face mark may be indicated by, for example, "normal (smile mark)", "abnormality (sadness mark)", or "notification (speech mark)".

Next, the information processing device 1 in the learning stage will be described with reference to FIG. For example, in the learning stage, the corresponding person image 30 (35) input from an input device (not shown) and one or a plurality of corresponding object images 40 to 43 are learned as a set. Here, learning refers to, for example, supervised learning. In the following description, the corresponding person image 30 (35) will be described as an example, but in addition to the corresponding person image 30 (35), for example, the corresponding person image, the corresponding person identification information, the corresponding person information, and the like may be used. , The target person including the responder and the subject, and the target person identification information 61 including the responder identification information and the respondent information.

FIG. 2 is a block diagram showing the configuration of the information processing apparatus 1 in the learning stage according to the present embodiment. In the learning stage, the information processing apparatus 1 includes a first trained model generation unit 9 and a second trained model generation unit 10.

The first trained model generation unit 9 generates the first trained model DB1 by training the first learning model DB1'with the scene ID and the corresponding person image 30 (35) as a pair. It is a program to do.

The first trained model generation unit 9 acquires the scene ID from the scene table TB1 with respect to the corresponding person image 30 (35), and acquires the model ID corresponding to the scene ID from the model table TB2.

The second trained model generation unit 10 designates a model ID and learns from the second learning model DB 2'with one or a plurality of chunk meta IDs and the corresponding object images 40 (41 to 43) as a pair. It is a program that generates the second trained model DB2 by making it.

The second learned model generation unit 10 acquires the content ID from the scene / content table TB4, which is a table in which the scene ID and the content ID are linked one-to-many, using the scene ID as a search key. Here, the scene ID that serves as a search key is associated with the corresponding person image 30 (35) that is paired with the corresponding object image 40 (41 to 43) to be processed.

The second learned model generation unit 10 acquires content from the content table TB3, which is a table in which the content ID and the content are linked one-to-one, using the content ID as a search key.

The second learned model generation unit 10 acquires the chunk ID from the content chunk table TB5, which is a table in which the content ID and the chunk ID are linked one-to-one or many, using the content ID as a search key.

The second learned model generation unit 10 acquires chunks from the chunk table TB7 using the chunk ID as a search key, and acquires a chunk meta ID from the chunk meta table TB6 using the chunk ID as a search key.

The second trained model generation unit 10 acquires the chunk meta value from the chunk meta table TB8 using the chunk meta ID as a search key. The chunk meta table TB8 is a table in which the chunk category ID, the chunk category name, and the chunk meta value are linked to the chunk meta ID on a one-to-one basis.

The chunk category ID uniquely indicates the chunk category name, which is the name of the category to which the chunk meta value belongs. The second trained model generation unit 10 refers to the corresponding object images 40 (41 to 43) and confirms that there is no problem in the acquired chunks, contents, and chunk meta values.

By determining a problematic value as an abnormal value and not using it for supervised learning, the second trained model generation unit 10 can generate a highly accurate trained model DB2, and the usage stage. The information processing apparatus 1 can perform highly accurate processing.

Next, the corresponding person image 30, the target person identification information 44, and the subject are generated by dividing the original image 20 and the original image 20 that are acquired by the user terminal 12 and processed as information by the information processing apparatus 1 using FIG. Corresponding object images 40 to 43 will be described. FIG. 3 is a diagram showing an original image 20, a corresponding person image 30, a target person identification image 44, and a plurality of corresponding object images 40 to 43 according to the present embodiment.

The original image 20, the corresponding person image 30, the target person identification image 44, and the plurality of corresponding object images 40 to 43 are displayed on, for example, the user terminal 12. Although FIG. 3 shows an example of being displayed at the same time, the original image 20, the corresponding person image 30, the target person identification image 44, and the plurality of corresponding object images 40 to 43 are separately displayed on the user terminal 12. You may.

The original image 20 captures the corresponding person 21, the target person identification image 44, and the corresponding objects 22 to 25. The size of the corresponding objects 22 to 25 is estimated based on information that does not change for each scene in the booth such as a desk. Further, the corresponding objects 22 to 25 include, like the corresponding object 24, information on the contents such as the attached photo 26, the internal text 27, and the signature 28, as well as code information such as a bar code and a two-dimensional code. Various coupons and the like may be obtained. Further, the target person identification image 44 may include, for example, the target person identification information 61, the face photograph 61a of the target person (for example, the corresponding person or the person to be dealt with), the name of the target person 61b, and the barcode 61c for identifying the target person. good.

Code information such as barcodes and two-dimensional codes, and various coupons are printed on paper media in advance, or may be displayed on the screen of the user terminal 12 of the respondent or the respondent 21, for example. good.

Next, the relationship between the respondent 21 and the plurality of respondents 22 to 25 will be described with reference to FIG. FIG. 4 is a diagram showing a tree structure in which the respondent 21 and the plurality of counterparts 22 to 25 are related according to the present embodiment. In addition to the corresponding person 21, for example, the corresponding person and the target person identification information (for example, the corresponding person identification information, the corresponding person identification information, etc.) may be used. In the following, the subject will be described in detail by taking the respondent 21 as an example.

As shown in FIG. 4, the image segmentation section 5 has a plurality of respondents 21 and a plurality of respondents 21 as a tree structure in which the respondent 21 is a root node and a plurality of counterparts 22 to 25 are leaf nodes or internal nodes. It is associated with the corresponding objects 22 to 25.

Further, the image dividing unit 5 further includes information such as an attached photograph 26, a text 27, and a signature 28, which are information included in at least one of the objects 22 to 25, as well as code information such as a barcode and a two-dimensional code. , Various coupons, etc. may be acquired and associated with the tree structure as a leaf node.

Next, the first trained model DB1 and the second trained model DB2 will be described with reference to FIG. FIG. 5 shows a first trained model DB1 and a second trained model DB2 according to the present embodiment.

The first trained model DB 1 includes a plurality of respondent images 30 (35), a plurality of scene IDs, and a plurality of respondent images 30 (35) generated by machine learning using a plurality of the trained model DB 1 as a pair of learning data. The connection between them is remembered. Here, machine learning is, for example, a convolutional neural network (CNN).

The association between the respondent image 30 (35) and the scene ID is specifically represented by the nodes indicated by circles in FIG. 5, the edges indicated by arrows, and the weighting factors set for the edges. It can be represented by a convolutional neural network. As shown in FIG. 5, the input of the corresponding person image 30 (35) to the first trained model DB1 is for each pixel such as pixels p1 and p2.

The second trained model DB2 is associated with the model ID on a one-to-one basis, and there are a plurality of them. Each of the second trained model DB2 is generated by machine learning using a plurality of object images 40 (41 to 43) and one or a plurality of chunk meta IDs as a pair of training data. The association between the plurality of object images 40 (41 to 43) and the plurality of one or a plurality of chunk meta IDs is stored. Here, machine learning is, for example, a convolutional neural network (CNN).

The association between the plurality of object images 40 (41 to 43) and the plurality of one or a plurality of chunk meta IDs is specifically the node indicated by a circle and the edge indicated by an arrow in FIG. It can be represented by a convolutional neural network represented by and the weighting factor set on the edge. As shown in FIG. 5, the input of the object image 40 (41 to 43) to the second trained model DB2 is for each pixel such as pixels p1 and p2.

Next, using FIG. 6, the scene table TB1, the model table TB2, the content table TB3, the scene content table TB4, the content chunk table TB5, the chunk meta table TB6, and the chunk, which are the information stored in the auxiliary storage device 11. The table TB7, the chunk meta table TB8, and the recommendation table TB9 will be described. FIG. 6 is a diagram showing information stored in the auxiliary storage device 11 according to the present embodiment.

The scene ID stored in the scene table TB1 or the like is a 3-digit hexadecimal number such as 0FD. The scene name stored in the scene table TB1 or the like is, for example, a grade inquiry or a career counseling.

The model ID stored in the model table TB2 or the like is represented by a two-character alphabetic character and a one-digit decimal number, for example, MD1. The content ID stored in the content table TB3 or the like is represented by a 5-digit hexadecimal number and a 2-digit decimal number, for example, 1B827-01. The content stored in the content table TB3 or the like is, for example, 1B827-01. A file name that is a content ID, such as txt, is indicated with an extension, and a pointer to the substance of the content is stored.

The chunk ID stored in the content chunk table TB5 or the like is represented by a 5-digit and 2-digit decimal number such as 82700-01. The chunk meta ID stored in the chunk meta table TB6 or the like is a 4-digit hexadecimal number such as 24FD.

The chunks stored in the chunk table TB7 are, for example, 1B827-01. It is indicated by a file name of the content corresponding to the target chunk and a one-digit decimal number such as pxt_0, and a pointer to a part of the substance of the content corresponding to the target chunk is stored.

The chunk summary stored in the chunk table TB7 is a document summarizing the contents of the chunk, for example, "Hello Work, ...". The hash value stored in the chunk table TB7 is a 15-digit hexadecimal number such as 564544d8f0b746e.

The chunk category ID stored in the chunk meta table TB8 is a 3-digit decimal number such as 394. The chunk category name stored in the chunk meta table TB8 is, for example, the size of the paper, the color of the paper, or the presence or absence of holes in the paper. The chunk meta values stored in the chunk meta table TB8 are, for example, A4, B4, white, blue, with holes on the sides, and without holes. The value of the chunk category ID and the chunk category name may be NULL.

The combination of chunk meta IDs stored in the recommendation table TB9 is (24FD, 83D9), (25FD), etc., and one or more chunk meta IDs are combined. The recommended object image stored in the recommendation table TB9 is, for example, IMG001. A pointer to an entity whose file name is indicated with an extension, such as jpg, is stored.

The shared object information stored in the recommendation table TB9 is, for example, IMG111. A pointer to an entity whose file name is indicated with an extension, such as jpg, is stored. Shared object information includes, for example, images, various video videos, texts such as documents and emails, online (online conferences, teleconferencing, videophones, etc.), applications for chats, link information, and the like. May be good. The co-owner information stored in the recommendation table TB9 stores, for example, the attributes and conditions of the other party with whom the information is shared, such as the target person and the inspector, and the information for identifying the co-owner, in association with each other.

The information processing device 1 refers to, for example, the co-owner information stored in the recommendation table 7, and shares the shared object information through the display area of the object model described later.

As shown in the scene content table TB4, the content chunk table TB5, and the chunk meta table TB6, the data structure of the work information has the chunk ID as the first layer and the chunk ID as the second layer. It has a hierarchical structure in which the content ID is a third layer and the scene ID is a fourth layer, which is the uppermost layer.

Next, the scene estimation function, chunk estimation function, and chunk output function will be described with reference to FIG. 7. FIG. 7 is a sequence diagram for explaining the scene estimation function, chunk estimation function, and chunk output function according to the present embodiment.

The information processing functions at the usage stage include a scene estimation function realized by the scene estimation process S60 described later, a chunk estimation function realized by the chunk estimation process S80 described later, and a chunk output realized by the chunk output process S100 described later. It consists of functions.

First, the scene estimation function will be explained. The image acquisition unit 4 included in the control unit 15 receives the original image 20 from the user terminal 12 (S1). Next, the image segmentation unit 5 included in the control unit 15 divides the original image 20 into the corresponding person image 30 and the corresponding object images 40 to 43.

The image segmentation unit 5 transmits the corresponding person image 30 to the scene estimation unit 6 and transmits the corresponding object images 40 to 43 to the chunk estimation unit 7. Next, the scene estimation unit 6 included in the control unit 15 inputs the corresponding person image 30 into the first trained model DB 1 (S2).

The first trained model DB 1 selects one or a plurality of scene IDs strongly associated with the received image 30, and selects one or a plurality of scene IDs for the scene estimation unit 6 (hereinafter, this is the first first). It may be called a scene ID list) (S3).

When the scene estimation unit 6 acquires the first scene ID list, it transmits it to the user terminal 12 as it is (S4). The user terminal 12 transmits to the scene estimation unit 6 whether or not there is a cache for each scene ID included in the first scene ID list (S5).

The user terminal 12 holds a table equivalent to the scene table TB1 with respect to the information processed in the past. The user terminal 12 searches the table held by the user terminal 12 using the scene ID of the received first scene ID list as a search key. The scene ID in which the search result is found has a cache, and the scene ID in which the search result cannot be found has no cache.

The scene estimation unit 6 has one or a plurality of scene IDs (hereinafter, referred to as a second scene ID) having no cache in the user terminal 12 among the respective scene IDs included in the first scene ID list received from the user terminal 12. The scene table TB1 is searched using (may be called a list) as a search key (S6).

The scene estimation unit 6 acquires, as a search result, a scene name corresponding to each scene ID included in the second scene ID list (hereinafter, this may be referred to as a scene name list) from the scene table TB1 (S7). ..

The scene estimation unit 6 transmits the acquired scene name list to the user terminal 12 as it is (S8). At the usage stage, the information processing apparatus 1 realizes a scene estimation function for estimating the scene of the person to be addressed image 30 by estimating the scene name in steps S1 to S8.

Next, the chunk estimation function will be explained. The user terminal 12 presents the received scene name list to the target person. The presentation of the scene name list is displayed, for example, on one side of the object model previously assigned by the target person. The subject selects, for example, one scene name from the presented scene name list. The user terminal 12 transmits the scene name selected by the target person to the chunk estimation unit 7 included in the control unit 15 (S9).

The chunk estimation unit 7 uses the scene ID corresponding to the scene name received from the user terminal 12 as a search key (S10), searches the model table TB2, and acquires the model ID (S11).

The chunk estimation unit 7 receives the corresponding object image 40 (41 to 43) from the image segmentation unit 5 (S12). The chunk estimation unit 7 designates one of the plurality of second trained model DB2s by the model ID acquired from the model table TB2, and designates the corresponding object images 40 (41 to 43) for the second learning. It is input to the completed model DB2 (S13).

The second trained model DB 2 selects one or a plurality of chunk meta IDs strongly associated with the object image 40 (41 to 43), and selects one or a plurality of the chunk estimation unit 7. A plurality of one or a plurality of chunk meta IDs (hereinafter, this may be referred to as a chunk meta ID list) are output (S14).

The chunk estimation unit 7 searches the chunk meta table TB6 using each one or a plurality of chunk meta IDs included in the chunk meta ID list as a search key (S15).

The chunk estimation unit 7 acquires one or a plurality of chunk IDs (hereinafter, this may be referred to as a first chunk ID list) from the chunk metatable TB6 as a search result (S16). The chunk estimation unit 7 transmits the acquired first chunk ID list to the user terminal 12 as it is (S17).

The user terminal 12 transmits to the chunk estimation unit 7 whether or not there is a cache for each chunk ID included in the first chunk ID list (S18). The user terminal 12 holds a table including a chunk ID column and a chunk summary column in the chunk table TB7 with respect to the information processed in the past.

The user terminal 12 searches the table held by the user terminal 12 using the chunk ID of the received first chunk ID list as a search key. Chunk IDs for which search results are found are cached, and chunk IDs for which search results are not found are cached.

The chunk estimation unit 7 has one or a plurality of chunk IDs (hereinafter, referred to as a second chunk ID) having no cache in the user terminal 12 among the respective chunk IDs included in the first chunk ID list received from the user terminal 12. The chunk table TB7 is searched using (which may be called a list) as a search key (S19).

The chunk estimation unit 7 acquires a chunk summary (hereinafter, this may be referred to as a chunk summary list) corresponding to each chunk ID included in the second chunk ID list as a search result from the chunk table TB7 (S20). .. The chunk estimation unit 7 transmits the acquired chunk summary list to the user terminal 12 as it is (S21).

At the usage stage, the information processing apparatus 1 realizes a chunk estimation function for estimating chunks of the corresponding object 22 (23 to 25) by estimating chunk summaries in steps S9 to S21.

Next, the chunk output function will be explained. The user terminal 12 presents the received chunk summary list to the target person. The chunk summary list presentation is displayed, for example, on one side of the object model pre-assigned by the subject. The subject selects, for example, one chunk summary from the presented chunk summary list. The user terminal 12 transmits the chunk summary selected by the target person to the chunk output unit 8 included in the control unit 15 (S22).

The chunk output unit 8 uses the chunk ID corresponding to the chunk summary received from the user terminal 12 as a search key (S23), searches the chunk table TB7, and acquires chunks (S24).

The chunk output unit 8 transmits the acquired chunk to the user terminal 12 as it is (S25). The user terminal 12 presents the received chunk to the user. The chunk presentation is displayed, for example, on one side of the object model pre-assigned by the subject. At the usage stage, the information processing apparatus 1 realizes a chunk output function for outputting chunks of the corresponding object 22 (23 to 25) by steps S22 to S25.

Next, the first trained model generation function and the second trained model generation function will be described with reference to FIG. FIG. 8 is a sequence diagram provided for explaining the first trained model generation function and the second trained model generation function according to the present embodiment.

The information processing functions at the learning stage include a first trained model generation function realized by the first trained model generation process and a second trained model generation function realized by the second trained model generation process. And consists of.

First, the first trained model generation function will be explained. The first trained model generation unit 9 included in the trained model generation unit 16 is a set of a scene name to be processed, a corresponding person image 30, and one or a plurality of corresponding object images 40 to 43. Is determined, and the scene table TB1 generated in advance is searched for in the scene table TB1 using the scene name as a search key (S31).

The first trained model generation unit 9 acquires the scene ID from the scene table TB1 as a search result (S32), and sets the corresponding person image 30 and the scene ID in the first learning model DB 1'as a pair. (S33).

Further, the first trained model generation unit 9 transmits the acquired scene ID to the model table TB2 and makes a model ID acquisition request (S34). The model table TB2 generates a model ID corresponding to the received scene ID and stores the combination of the scene ID and the model ID.

Next, the first trained model generation unit 9 acquires the model ID from the model table TB2 (S35). In the learning stage, the information processing apparatus 1 realizes the first trained model generation function of generating the first trained model DB1 by steps S31 to S35.

Next, the second trained model generation function will be described. The second trained model generation unit 10 included in the trained model generation unit 16 uses the scene ID received by the first trained model generation unit 9 in step S32 as a search key to generate a scene content table in advance. Search for TB4 (S36).

The second learned model generation unit 10 acquires the content ID from the scene content table TB4 as a search result (S37), and searches the content table TB3 generated in advance using the acquired content ID as a search key (S38). ..

The second learned model generation unit 10 acquires the content from the content table TB3 as a search result (S39), and searches the content chunk table TB5 generated in advance using the content ID acquired in step S37 as a search key (S). S40).

The second trained model generation unit 10 acquires the chunk ID from the content chunk table TB5 as a search result (S41), and searches the chunk table TB7 generated in advance using the acquired chunk ID as a search key (S42). ..

The second trained model generation unit 10 acquires chunks from the chunk table TB7 as a search result (S43), and searches the chunk metatable TB6 generated in advance using the chunk ID acquired in step S41 as a search key (S). S44).

The second trained model generation unit 10 acquires one or a plurality of chunk meta IDs from the chunk meta table TB6 as search results (S45), and pre-generates each acquired meta ID for chunks as a search key. The meta table TB8 for chunks is searched (S46).

The second trained model generation unit 10 acquires the chunk meta value corresponding to each chunk meta ID as a search result from the chunk meta table TB8 (S47).

The second trained model generation unit 10 checks whether there is a problem in the content acquired in step S39, the chunk acquired in step S43, and the respective chunk meta values acquired in step S47, with the corresponding person image 30 and the corresponding. Confirmation is performed with reference to the object images 40 to 43.

For example, the second trained model generation unit 10 confirms by referring to the facial expressions of the person to be corresponded 21 and the document names described in the objects to be dealt with 22 to 25. The second trained model generation unit 10 determines, for example, the facial expression of the corresponding person 21 from the corresponding person image 30, and assigns the document names described in the corresponding objects 22 to 25 from the corresponding object images 40 to 43. judge.

As a result of reference, if there is a problem that it is clear that the content, chunks, and meta values for chunks are information about documents that are clearly different from the documents captured in the corresponding object images 40 to 43. End the processing for the target set.

Next, the second trained model generation unit 10 sets a pair of the model ID, the corresponding object images 40 (41 to 43), and one or a plurality of chunk meta IDs in the second learning model DB 2'. (S48). In the learning stage, the information processing apparatus 1 realizes the second trained model generation function of generating the second trained model DB 2 by steps S36 to S48.

Next, information processing at the usage stage will be described with reference to FIG. FIG. 9 is a flowchart showing a processing procedure of information processing in the usage stage according to the present embodiment. Information processing in the usage stage is composed of a scene estimation process S60, a chunk estimation process S80, and a chunk output process S100.

First, the scene estimation process S60 will be described. The scene estimation process S60 is composed of steps S61 to S67. When the scene estimation unit 6 receives the corresponding person image 30 (35) from the image segmentation unit 5 (S61), the scene estimation unit 6 inputs the corresponding person image 30 (35) into the first trained model DB 1 (S62).

The scene estimation unit 6 acquires the first scene ID list as output from the first trained model DB 1 (S63), transmits the first scene ID list to the user terminal 12 as it is, and determines whether or not there is a cache. Contact 12 (S64).

When all the response results from the user terminal 12 have a cache (S65: NO), the scene estimation process S60 ends and the chunk estimation process S80 starts. When even one response result from the user terminal 12 has no cache (S65: YES), the scene estimation unit 6 acquires the scene name list from the scene table TB1 (S66) and transmits it to the user terminal 12 as it is (S67). ), The scene estimation process S60 ends.

Next, the chunk estimation process S80 will be described. The chunk estimation process S80 is composed of steps S81 to S88. The chunk estimation unit 7 receives the scene name selected by the target person from the user terminal 12 (S81).

Upon receiving the scene name from the user terminal 12, the chunk estimation unit 7 acquires the model ID from the model table TB2 (S82). Next, the chunk estimation unit 7 designates one of the plurality of second learned models DB 2 by the model ID, and designates the corresponding object images 40 (41 to 43) received from the image division unit 5. It is input to the trained model DB2 of 2 (S83).

The chunk estimation unit 7 acquires a chunk meta ID list as an output from the second trained model DB 2 (S84), and acquires a first chunk ID list from the chunk meta table TB6 (S85). Next, the chunk estimation unit 7 transmits the first chunk ID list to the user terminal 12 as it is, and inquires the user terminal 12 whether or not there is a cache (S86).

When all the response results from the user terminal 12 have a cache (S86: NO), the chunk estimation process S80 ends and the chunk output process S100 starts. When even one response result from the user terminal 12 has no cache (S86: YES), the chunk estimation unit 7 acquires a chunk summary list from the chunk table TB7 (S87) and transmits it to the user terminal 12 as it is (S88). ), The chunk estimation process S80 ends.

Next, the chunk output process S100 will be described. The chunk output process S100 is composed of steps S101 to S103. The chunk output unit 8 receives the chunk summary selected by the target person from the user terminal 12 (S101).

When the chunk summary is received from the user terminal 12, the chunk output unit 8 acquires the chunk from the chunk table TB7 (S102) and transmits it to the user terminal 12 as it is (S103), and the chunk output process S100 ends.

Next, the processing procedure of information processing in the display stage will be described with reference to FIG. FIG. 10 is a flowchart showing information processing in the display stage according to the present embodiment. The display on the display unit 14 is composed of the display process S110.

The display process S110 is composed of steps S111 to S114. The display unit 14 includes an object model specifying unit 15, for example, acquiring object model information stored in advance (S111) and acquiring information on each display area constituting the object model (S112). Specifically, for the information of each display area, for example, the object model specifying unit 15 specifies the recommended image output by the recommended image output unit and the object model for displaying the recommended information. The object model specifying unit 15 identifies the object model by associating the scene and chunk with the object model ID that uniquely indicates the object model.

The object model is specified, for example, with reference to the object model table TB10, and is specified based on various information such as a scene ID, a chunk ID, an area ID such as a target person or an information sharer, or a role ID. You may.

The display unit 14 is not assigned a display area of the object model based on various information such as, for example, the attribute, type, amount of information of the information presented to the user terminal, the type of the target person, the operation of the target person, and the like. (S113: NO) displays information other than the display target and default display. On the other hand, when the display area of the object model is allocated (S113: YES), various information to be displayed is assigned to each display area and displayed (S114).

Allocation of various information (objects) to be displayed to each display area of the object model refers to the object allocation table TB11, and is assigned to the display area based on, for example, a scene ID, a chunk ID, a role ID, and the like. .. The display area is allocated based on, for example, the display area information identified by the object model ID and the display area ID.

The display unit 14 displays the recommendation image and the recommendation information output by the recommendation image output unit in the display area of any of the plurality of display areas included in the object model specified by the object model identification unit 15 with the target person. It is assigned and displayed as a state that can be shared with the co-owner, and the display process S110 ends.

Next, information processing in the learning stage will be described with reference to FIG. FIG. 11 is a flowchart showing a processing procedure of information processing in the learning stage according to the present embodiment. Information processing in the learning stage is composed of a first trained model generation process S120 and a second trained model generation process S140.

First, the first trained model generation process S120 will be described. The first trained model generation process S120 is composed of steps S121 to S124. When the first trained model generation unit 9 determines a set of the scene name, the corresponding person image 30 (35), and one or more corresponding object images 40 (41 to 43), the scene name is determined. Search the scene table TB1 as a search key (S121).

The first learned model generation unit 9 acquires the scene ID from the scene table TB1 as a search result (S122), and puts the scene ID and the corresponding person image 30 (35) in the first learning model DB 1'. Learn as a pair (S123).

Next, the first trained model generation unit 9 transmits the scene ID acquired in step S122 to the model table TB2, makes a model ID acquisition request, and acquires the model ID (S124).

Next, the second trained model generation process S140 will be described. The second trained model generation process S140 is composed of steps S141 to S150. The second learned model generation unit 10 searches the scene content table TB4 using the scene ID acquired in step S122 as a search key, and acquires the content ID (S141).

The second learned model generation unit 10 searches the content table TB3 using the acquired content ID as a search key and acquires the content (S142). Further, the second learned model generation unit 10 searches the content chunk table TB5 using the acquired content ID as a search key, and acquires the chunk ID (S143).

Further, the second trained model generation unit 10 searches the chunk table TB7 using the acquired chunk ID as a search key and acquires chunks (S144). Further, the second learned model generation unit 10 searches the chunk meta table TB6 using the acquired chunk ID as a search key, and acquires one or a plurality of chunk meta IDs (S145).

Further, the second trained model generation unit 10 searches the chunk meta table TB8 using each of the acquired one or a plurality of chunk meta IDs as a search key, and the chunk meta value corresponding to each chunk meta ID. (S146).

The second trained model generation unit 10 checks whether there is a problem with the content acquired in step S142, the chunk acquired in step S144, and the respective chunk meta values acquired in step S146 (corresponding person image 30 (35)). And the corresponding object images 40 (41 to 43) are referred to for confirmation (S147).

If there is a problem as a result of confirmation (S148: NO), the information processing in the learning stage regarding the set being processed ends. If there is no problem as a result of confirmation (S148: YES), the second trained model generation unit 10 has a model ID, one or more chunk meta IDs, and a corresponding object image in the second learning model DB 2'. 40 (41 to 43) and 40 (41 to 43) are learned as a pair (S149), and the information processing in the learning stage regarding the set being processed is completed.

As described above, the chunk that divides or suggests the work information by the information processing apparatus 1 according to the present embodiment is presented via the user terminal 12. Therefore, it is possible to present the required amount of information by appropriately setting chunks. Also, if the chunk is information that suggests the entire document, there is no need to reconstruct the information on a large scale.

(The present embodiment: the second embodiment)
Next, the operation of the information processing system 100 at the utilization stage will be described with reference to FIGS. 12, 17 and 18. FIG. 12A is a schematic diagram showing an example of the operation of the information processing system 100, and FIG. 12B is a diagram showing an original image, a target person image, and a plurality of objects to be imaged in the information processing system 100. Is. 17 (a) to 17 (h) are diagrams showing display patterns in the target person in the information processing system 100, and FIGS. 18 (a) to 18 (b) are user terminals 12 in the information processing system 100. It is a schematic diagram which shows an example of the display of.

In the schematic diagram showing an example of the operation of the information processing system 100 shown in FIG. 12 (a), for example, the target person (correspondent, correspondent, co-owner, trainee, etc.) 50a with respect to the corresponding object 60 in the manufacturing area. The case where the work information about the work is output when the work is performed is shown.

Here, the information processing system 100 is, for example, via a user terminal 12 (for example, a head-mounted display) worn by the target person 50a, for example, an employee ID card 61 which is the target person identification information, and a corresponding object 60 which performs work. Get an image. In addition to the employee ID card 61, the target person identification information may be, for example, an image of the subject's face, fingerprint, palmistry, vein, or the like, and may be unique information that can identify the target person.

The information processing system 100 acquires the target person identification information of the employee ID card 61 taken by the target person 50b if, for example, the target person 50b who collaborates with the target person 50b is in the same work area in addition to the target person 50a in the manufacturing area. You may try to do it. The information processing system 100 acquires, for example, the target person identification information of the employee ID card 61 of the target person 50b and the image of the corresponding object 60 captured from the target person 50b side from the target person 50b.

In the information processing system 100, for example, when a plurality of target persons 50a, target persons 50b, and the like collaborate in a manufacturing area, for example, an image taken by a camera of each user device 12 is used as a corresponding object in the collaborative work. It may be specified as an image of the image or divided. The information processing system 100 may, for example, determine the divided image and search for the recommended recommended object image and the shared shared object information.

According to the recommendation image output unit 13, the information processing system 100 needs to share information in the work of the target person, for example, an image of a corresponding object that is not captured in the original image but is presumed to be necessary in the original image. It is possible to output a recommended image including information on the presumed object to be handled and the work of the target person, and it is possible to prevent forgetting to work.

The manufacturing area is connected to the customer / support area, which is the area of the customer / trainer 51, via a communication network such as the Internet, and the monitoring is, for example, the area of the inspector 52 in which the target person 50a monitors the work. Connected to the area. The target person 50a outputs information necessary for one work from a viewpoint according to each position to the customer / trainer 51 and the inspector 52 from various aspects, and has a plurality of different positions, work places, and work hours. Enables information sharing among workers.

In the information processing system 100, the target person 50a shares information in the customer / trainer 51 and the inspector 52 via the above-mentioned object model. The information processing system 100 searches for, for example, a recommended recommended object image and shared shared object information by the recommendation image output unit 13, and although it is not captured in the original image, it is originally necessary. It outputs a recommended image including an image of the object to be presumed to be, an object to be presumed to need information sharing in the work of the subject, and information on the work of the subject. The recommendation image output unit 13 allocates and outputs to a plurality of display areas of the object model.

FIG. 12B is a diagram showing an original image, a target person image, and a plurality of objects to be imaged in the information processing system 100. The information processing system 100 acquires, for example, the target person identification information 70 regarding the target person and the corresponding object 60 image taken by the user terminal 12 worn by the target person 50a by the image acquisition unit 4, and causes the auxiliary storage device 11 to acquire the target person identification information 70 and the corresponding object 60 image. It is associated with each and stored.

The image stored in the auxiliary storage device 11 may be, for example, an image of a target person or target person identification information, for example, an image of an employee ID card 61. The employee ID card 61 may include, for example, a face image 61a, a name 61b, and code information 61c of a person who performs work. Further, the image stored in the auxiliary storage device 11 includes, for example, an image of the corresponding object 60. The image of the object 60 may include, for example, images of the

parts

60a, 60b, and 60c constituting the object 60.

The original image captured by the image acquisition unit 4 is divided by the image segmentation unit 5. The image segmentation unit 4 may be divided into parts 60a to 60c, for example, after the image acquisition unit 4 acquires an image of the object 60. The target person identification information may be, for example, information for identifying the target person, and may be, for example, an image of the employee ID card 61. When the target person identification information is, for example, an employee ID card 61, the face image 61a, the name 61b, and the code information 61c of the target person who performs the work may be included.

The images divided by the original image segmentation unit 4 acquired by the image acquisition unit 4 are stored as

images

70, 71, 71a to 71c in association with each other in the auxiliary storage device 11.

Here, an example of the display of the user terminal 2 in the present embodiment will be described with reference to FIG. FIG. 17 shows various display contents assigned and displayed by the object model specifying unit 15 to a plurality of display areas of the object model specified by the object model specifying unit 15 in the user terminal 2.

FIG. 17A is an example in which a plurality of scene candidates estimated by the scene estimation unit 6 are displayed in one display area of the object model. FIG. 17B is an example in which content / difference information is displayed in the display area as, for example, an image or information associated with a chunk ID. In FIG. 17C, for example, the user terminal 12 is a device such as a smartphone, the image information of the user's viewpoint is switched to the rear camera, and the object to be photographed is photographed as a second image, and the photographed object is displayed in the display area. It is an example displayed in. In FIG. 17D, for example, a skilled person (trainer) photographs a target person (trainee) to work on a corresponding object, and a work checklist displayed based on the second image is displayed in the display area. It is an example to be done. FIG. 17E is an example of being displayed in the display area as an image of a corresponding object recorded by, for example, a user terminal end 12 of a skilled person (trainer). FIG. 17 (f) is an example in which, for example, an image for confirming the behavior of a target person performing a work from a bird's-eye view and related information are displayed together in a display area. FIG. 17 (g) is an example in which the work information for generating the expert / AI learning data when acquiring the learning information recorded by the expert (trainer) is displayed in the display area. FIG. 17H is an example in which related moving image information and origin information are displayed in the display area as related narrative information associated with the chunk ID, for example.

In the display area of the specified object model, for example, each reference information displayed in FIGS. 17A to 17H is displayed as a recommended image. For example, the scene estimation unit 6 selects a scene and estimates chunks. In the scene of selecting chunks by the part 7, FIG. 17A may be displayed in front of the subject. Further, the display area of the object model may be rotated based on the work content, work status, etc. of the target person, and more important information, attention information, and the like may be preferentially displayed.

Next, FIGS. 18A to 18B show the display contents on the user terminal 12 by the information processing system 100. In the display example shown in FIG. 18A, for example, the user terminal 12 is a smartphone and is displayed as a flat display. In the display example shown in FIG. 18A, for example, chunks and recommendations are made in the display areas 80a of the display screen 80 of the target person (correspondent) such as a smartphone or tablet, and in the respective display areas of the specified

object models

80b and 80c. This is an example in which various information such as images are displayed.

In the display example shown in FIG. 18A, for example, the image of the object model may be shared between different user terminals 12. As a result, for example, even when multiple target persons work together, when each target person needs it, information narrowed down to each target person, useful information that the target person does not notice, and even among the target persons , It will be possible to share various information with customers / trainers, inspectors, etc. at other bases. Thereby, for example, it can be presented to the target person as a common operation method and UI, and the effectiveness of information provision and the usefulness of information can be improved.

Next, in the display example shown in FIG. 18B, for example, the user terminal 12 is a personal computer or the like, and the target person is an inspector. In this case, for example, on the display screen 81, a display area 81a for displaying a bird's-eye view of the target person working on the object to be handled, a display area 81b for displaying a viewpoint image of a skilled person in charge of the skilled person, and the target person. A display area 81c for selecting an object model for transmitting information and a display area 81d for transmitting an alert to a target person (for example, a trainer, a trainee, etc.) are displayed. As a result, when the target person needs it, the necessary information, shared information, related information, etc. are presented to the target person (correspondent), etc. from a viewpoint and role different from the information narrowed down to the target person. Is possible. Furthermore, the effectiveness of information provision and the usefulness of information can be further improved.

Further, according to the present embodiment, the object model specifying unit 15 may allocate, for example, schematic information to the display area. Schematized information includes, for example, figures and illustrations showing facial expressions such as "smile", "anxiety", and "tension" that simplify human facial expressions, and "caution" and "warning" for the work situation of the subject. Words and messages, as well as light emission states such as red, blue, and yellow may be displayed as indicator lights and the like.

By using the model table TB2, even if the relationship between the first trained model DB1 and the second trained model DB2 changes, it is possible to deal with it simply by changing the model table TB2, which is excellent in maintainability. Can provide the equipment.

When the model table TB2 is not used, it is necessary to regenerate the trained model DB2 when the relationship between the first trained model DB1 and the second trained model DB2 changes.

In the present embodiment, the image acquisition unit 4, the image division unit 5, the scene estimation unit 6, the chunk estimation unit 7, the chunk output unit 8, the first trained model generation unit 9, and the second trained model generation unit 9 are used. The 10 and the recommendation image output unit 13 are programmed, but the program is not limited to this, and a logic circuit may be used.

Further, the image acquisition unit 4, the image division unit 5, the scene estimation unit 6, the chunk estimation unit 7, the chunk output unit 8, the first trained model generation unit 9, the second trained model generation unit 10, and the recommendation image output unit. 13, 1st trained model DB1, 1st training model DB1', 2nd trained model DB2, 2nd learning model DB2', scene table TB1, model table TB2, content table TB3, scene content table TB4, content chunk table TB5, chunk meta table TB6, chunk table TB7, chunk meta table TB8, recommendation table TB9, object table TB10, object allocation table TB11, annotation table TB12, attention table TB13, camera table TB14, roll The table TB15 is not mounted on one device, but may be distributed and mounted on a plurality of devices connected by a network.

Further, in the learning stage shown in FIGS. 8 and 11 described above, a case where the first trained model and the second trained model are associated and generated has been described, but the present invention is not limited to this, and the first method is not limited to this. The trained model DB1 and the second trained model DB2 may be generated separately.

When the first trained model DB1 and the second trained model DB2 are generated separately, for example, when the scene is an existing one and only the content is added, it is not necessary to learn about the scene.

In the present embodiment, the case where a plurality of second trained model DB2s are used has been described, but the present invention is not limited to this, and only one second trained model DB2 may be used. Further, in the present embodiment, the case of displaying the image of the corresponding object that is presumed to be originally necessary has been described, but the present invention is not limited to this, and a part of the corresponding object that is presumed to be originally necessary is displayed. May be displayed. Further, in the present embodiment, a corresponding object or a part of the corresponding object, which is presumed to be originally unnecessary, may be suggested.

The information processing apparatus 1 of the present embodiment is composed of a tree structure associated with the image dividing unit and values output from the first trained model DB1 and the second trained model DB2 in the usage stage. By comparing with the hierarchical structure, excess / deficiency points may be determined.

In the present embodiment, for example, the recommendation image and the recommendation information assigned to the plurality of display areas of the object model by the display unit 14 are linked to the scene information and the scene information indicating the contents of the scene estimated by the scene estimation unit 6. Work information related to the work performed by the target person, work check information indicating the work process related to the work performed by the target person linked to the work information, chunk information linked to the work information, difference information of the work content related to the work, hand by the work trainer Includes at least one of model information showing the content of this work, work information showing the work video of the work scene of the target person, or instruction information shown according to the difference in work between the model information and the work information. You may do so.

In the present embodiment, for example, the display unit 14 may output the object model in the vicinity of the corresponding object in the virtual display space of the target person. Further, the display unit 14 may output the object model by fixing the display position in the vicinity of the corresponding object in the virtual display space of the target person.

In the present embodiment, for example, the object model specifying unit specifies the object model based on at least one of the skill information of the target person, the spatial information for performing the work, the characteristic information of the corresponding object, or the work level information for the work. You may do so.

In the present embodiment, for example, the object model specifying unit may specify one or more object models having two or more display areas. Further, the object model identification unit is in at least one of rotation display, enlarged display, reduced display, protrusion display, vibration display, state display, discoloration display, and shading display based on the state of the work performed by the subject. The object model may be displayed.

1 ... Information processing device, 2 ... Central processing unit, 3 ... Main storage device, 4 ... Image acquisition unit, 5 ... Image division unit, 6 ... Scene estimation unit, 7 ... Chunk estimation unit, 8 ...... Chunk output unit, 9 ... 1st trained model generation unit, 10 ... 2nd trained model generation unit, 11 ... auxiliary storage device, 12 ... user terminal, 13 ... recommended image output unit , 14 …… Display part, 15 …… Object model identification part, 100 …… Information processing system ……

Claims

An information processing device that outputs work information that is information about the work performed by the responder.
An image acquisition unit that acquires an original image that is an image including a target person including at least one of the corresponding person and the corresponding target person and a plurality of corresponding objects supported by the corresponding person.
An image segmentation unit that divides the original image into an image of the subject captured by the subject and a plurality of images of the corresponding objects captured by each of the objects.
Scene estimation for estimating the scene using the first trained model in which the relationship between the subject image and the scene ID uniquely indicating the scene performed by the correspondent is stored. Department and
Between the plurality of objects to be imaged, a chunk ID uniquely indicating a chunk that is information that divides or suggests the work information, and one or a plurality of chunk meta IDs associated with one-to-one. A chunk estimator that estimates the chunk using one of a plurality of second trained models in which the associations in are stored.
The chunk output section that outputs the chunk and
One of the plurality of second trained models is recommended as a search key using a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID in a one-to-one manner. A recommendation image output unit that searches for an image and outputs a recommendation image that is an image of the corresponding object that is not captured in the original image but is presumed to be necessary in the original image.
A display unit that allocates and displays the chunk output by the chunk output unit and the recommended image output by the recommendation image output unit to the display area of an object model having a plurality of display areas.
The chunk estimation unit is selected by using the model ID, and the chunk meta ID is an information processing device that uniquely indicates a chunk meta value which is information about the property of the corresponding object.
The image dividing unit associates the target person with the plurality of the corresponding objects as a tree structure in which the target person is a root node and the plurality of the corresponding objects are leaf nodes or internal nodes. Item 1. The information processing apparatus according to Item 1.
The information processing apparatus according to claim 2, wherein the image segmentation unit further acquires information contained in at least one of the corresponding objects and associates it with the tree structure as a leaf node.
The image of the target person acquired by the image acquisition unit is the target person identification information for identifying the target person.
The information processing apparatus according to claim 1.
The scene estimation unit uses the first trained model in which the association between the target person identification information and the scene ID uniquely indicating the scene performed by the correspondent is stored. Further estimating the scene,
The chunk estimation unit is among a plurality of second trained models in which the association between the plurality of object images, the chunk ID, and the plurality of chunk meta IDs is stored. One is used to further estimate the chunk associated with the subject identification information.
The recommendation image output unit searches for one of the plurality of second trained models by searching for a model ID associated with a scene ID on a one-to-one basis and a combination of one or a plurality of chunk meta IDs. As a method, the shared object information is searched, and the image of the object that is presumed to need to be shared in the work and the recommendation image including the information about the work of the target person are further output.
The information processing apparatus according to claim 1.
The display unit specifies the scene, the chunk, and the object model ID that uniquely indicates the object model, in which the scene, the chunk, and the object model that displays the recommended image and the recommended information output by the recommended image output unit are specified. To further provide an object model identification part that identifies the object model by associating it.
The information processing apparatus according to claim 1.
The display unit displays the recommendation image and the recommendation information output by the recommendation image output unit in any display area of a plurality of display areas included in the object model specified by the object model identification unit. Assigning as a state that can be shared between the target person and the co-owner who shares information with the target person,
The information processing apparatus according to claim 1 or 6.
The recommendation image output unit identifies a collaborator who collaborates with the responder, a trainer who is the leader of the responder, and an inspector who monitors the responder. Searching as a sharer,
4. The information processing apparatus according to claim 4.
It is an information processing method performed by an information processing device that outputs work information that is information about the work performed by the responder.
A first step of acquiring an original image which is an image including a target person including at least one of the corresponding person and the corresponding target person and a plurality of corresponding objects to which the corresponding person corresponds.
A second step of dividing the original image into an image of the subject captured by the subject and a plurality of images of the subject captured by each of the objects.
A third estimation of the scene using the first trained model in which the association between the subject image and the scene ID uniquely indicating the scene performed by the correspondent is stored. Steps and
Between the plurality of objects to be imaged, a chunk ID uniquely indicating a chunk that is information that divides or suggests the work information, and one or a plurality of chunk meta IDs associated with one-to-one. A fourth step of estimating the chunk using one of a plurality of second trained models in which the associations in the are remembered.
The fifth step of outputting the chunk and
One of the plurality of second trained models is recommended as a search key using a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID in a one-to-one manner. The sixth step of searching the image and outputting the recommended image which is the image of the corresponding object which is not captured in the original image but is presumed to be necessary in the original image.
Each of the chunk and the recommended image is provided with a seventh step of allocating and displaying each of the recommended images in the display area of the object model having a plurality of display areas.
One of the plurality of second trained models is selected using the scene ID and the model ID associated with one-to-one, and the chunk meta ID is information regarding the property of the corresponding object. An information processing method that uniquely indicates the chunk meta value.
An information processing system that outputs work information that is information about the work performed by the responder.
The corresponding person corresponds to the target person image including at least one of the corresponding person and the target person to which the corresponding person corresponds, and the target person identification information for identifying the target person. An image acquisition means for acquiring an original image which is an image including a plurality of objects to be supported,
An image segmentation means for dividing the original image into an image of the target person and a plurality of images of the corresponding objects in which the corresponding objects are captured.
Scene estimation for estimating the scene using the first trained model in which the relationship between the subject image and the scene ID uniquely indicating the scene performed by the correspondent is stored. Means and
Between the plurality of objects to be imaged, a chunk ID uniquely indicating a chunk that is information that divides or suggests the work information, and one or a plurality of chunk meta IDs associated with one-to-one. A chunk estimation means for estimating the chunk using one of a plurality of second trained models in which the associations in the are stored.
Chunk output means for outputting the chunk and
One of the plurality of second trained models is recommended as a search key using a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID in a one-to-one manner. The image of the corresponding object and the shared corresponding object information to be shared are searched, and the image of the corresponding object, which is not captured in the original image but is presumed to be originally necessary, and the information in the work of the target person. A recommendation image output means for outputting a recommendation image including information on the object to be shared and the work of the target person, which is presumed to need to be shared.
A display means for allocating and displaying the chunk output by the chunk output means and the recommended image output by the recommended image output means to the display area of an object model having a plurality of display areas.
The chunk estimation means is selected by using the model ID, and the chunk meta ID uniquely indicates a chunk meta value which is information about the property of the corresponding object.