WO2022124419A1 - Appareil de traitement d'informations, procédé de traitement d'informations et système de traitement d'informations - Google Patents

Appareil de traitement d'informations, procédé de traitement d'informations et système de traitement d'informations Download PDF

Info

Publication number
WO2022124419A1
WO2022124419A1 PCT/JP2021/045713 JP2021045713W WO2022124419A1 WO 2022124419 A1 WO2022124419 A1 WO 2022124419A1 JP 2021045713 W JP2021045713 W JP 2021045713W WO 2022124419 A1 WO2022124419 A1 WO 2022124419A1
Authority
WO
WIPO (PCT)
Prior art keywords
chunk
image
information
scene
model
Prior art date
Application number
PCT/JP2021/045713
Other languages
English (en)
Japanese (ja)
Inventor
聡 黒田
Original Assignee
株式会社 情報システムエンジニアリング
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社 情報システムエンジニアリング filed Critical 株式会社 情報システムエンジニアリング
Priority to JP2022568365A priority Critical patent/JPWO2022124419A1/ja
Publication of WO2022124419A1 publication Critical patent/WO2022124419A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"

Definitions

  • the present invention relates to an information processing apparatus, an information processing method and an information processing system.
  • a rule describing the judgment conditions of the work target or the work situation is generated based on the manual describing the work procedure, contents, points to be noted or other matters, and the worker. Recognizes the work target and work status based on the sensor information from the device worn by the user, and outputs work support information based on the recognition result of the generated rule and recognition means.
  • Patent Document 1 the information stored as a document such as a manual can only be searched for each document. For example, if you want to search a document paragraph by paragraph, you need to reconstruct the document into structured information. Reconstruction of all documents to be searched is often not realistic in terms of cost-effectiveness, and information on a document-by-document basis often browses unnecessary information, so that the viewer of the document can quickly browse. There is a problem that it may not be possible to deal with it.
  • One aspect of the embodiment of the present invention is information processing that presents the required amount of information to the responder and the collaborator when the responder needs it, without reconstructing the information on a large scale. It is an object of the present invention to provide an apparatus, an information processing method and an information processing system.
  • An information processing device that outputs work information that is information about work performed by a responder, and is a plurality of objects corresponding to the responder and a target person including at least one of the respondents to which the responder corresponds.
  • An image acquisition unit that acquires an original image that is an image including the corresponding object, a target image obtained by dividing the original image and captured by the target person, and a plurality of corresponding object images obtained by capturing each corresponding object.
  • the scene ID that uniquely indicates the scene that the correspondent performs, and the scene ID are stored.
  • a scene estimation unit that estimates a scene, a plurality of object images, a chunk ID that uniquely indicates a chunk that is information that divides or suggests work information, and one or a plurality of chunks that are associated with one-to-one.
  • a chunk estimator that estimates chunks and a chunk output unit that outputs chunks using one of a plurality of second trained models in which the association between the meta IDs is stored.
  • One of a plurality of second trained models is recommended as a search key using a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID in a one-to-one manner.
  • a recommendation image output unit that searches for an image and outputs a recommended image that is an image of a corresponding object that is not captured in the original image but is presumed to be necessary, and chunks and chunks output by the chunk output unit.
  • Each of the recommended images output by the recommended image output unit is provided with a display unit for allocating and displaying each of the recommended images to each surface of the object model having a plurality of display areas, and the chunk estimation unit is a plurality of second trained models. One of them is selected using the scene ID and the model ID associated with one-to-one, and the chunk meta ID uniquely indicates the chunk meta value which is information on the property of the corresponding object.
  • a step and one of a plurality of second trained models are recommended as a search key using a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID in a one-to-one manner.
  • Each is provided with a seventh step of assigning and displaying each side of an object model having a plurality of display areas, and one of the plurality of second trained models has a one-to-one correspondence with the scene ID.
  • the chunk meta ID which is selected using the attached model ID, provides an information processing method that uniquely indicates the chunk meta value which is information about the property of the corresponding object.
  • An information processing system that outputs work information that is information about work performed by a responder, and identifies the responder, a target person including at least one of the respondents to which the responder corresponds, and the target person.
  • An image acquisition means for acquiring an original image which is an image including at least one of the target person identification information and a plurality of corresponding objects to which the corresponding person corresponds, and the target person using the original image.
  • An image dividing means for dividing an image into a plurality of images of the corresponding objects captured by each of the corresponding objects, an image of the target person, and a scene ID uniquely indicating a scene in which the corresponding object is performed.
  • the scene estimation means for estimating the scene, the plurality of objects to be imaged, and the information obtained by dividing or suggesting the work information.
  • One of a plurality of second trained models in which the association between a chunk ID uniquely indicating a certain chunk and one or more chunk meta IDs associated with one-to-one is stored.
  • One of the chunk estimation means for estimating the chunk, the chunk output means for outputting the chunk, and the plurality of second trained models has a one-to-one correspondence with the scene ID.
  • the attached model ID and a combination of one or more chunk meta IDs as a search key, the recommended recommended object image and the shared shared object information are searched, and the original image is captured.
  • the recommendation image output means for outputting the recommendation image, the chunk output by the chunk output means, and the recommendation image output by the recommendation image output means are assigned to the display area of the object model having a plurality of display areas.
  • the chunk estimation means is selected by using the model ID, and the chunk meta ID uniquely indicates a chunk meta value which is information about the property of the corresponding object. Provides an information processing system.
  • an information processing device that presents the required amount of information to the responder when the responder needs it, without reconstructing the information on a large scale.
  • Information processing method and information processing system can be realized.
  • FIG. 1 is a block diagram showing a configuration of an information processing apparatus at a utilization stage according to the present embodiment.
  • FIG. 2 is a block diagram showing a configuration of an information processing apparatus in the learning stage according to the present embodiment.
  • FIG. 3 is a diagram showing an original image, a subject image, and a plurality of objects to be imaged according to the present embodiment.
  • FIG. 4 is a diagram showing a tree structure which is a relationship between a subject image and a plurality of objects to be imaged according to the present embodiment.
  • FIG. 5 is a diagram showing a first trained model and a second trained model according to the present embodiment.
  • FIG. 6 is a diagram showing information stored in the auxiliary storage device according to the present embodiment.
  • FIG. 7 is a sequence diagram for explaining the scene estimation function, chunk estimation function, and chunk output function according to the present embodiment.
  • FIG. 8 is a sequence diagram provided for explaining the first trained model generation function and the second trained model generation function according to the present embodiment.
  • FIG. 9 is a flowchart showing a processing procedure of information processing in the usage stage according to the present embodiment.
  • FIG. 10 is a flowchart showing a processing procedure of information processing in the display stage according to the present embodiment.
  • FIG. 11 is a flowchart showing a processing procedure of information processing in the learning stage according to the present embodiment.
  • FIG. 12A is a schematic diagram showing an example of the operation of the information processing system according to the second embodiment, and FIG.
  • FIG. 12B is an original image, a subject image, and a plurality of objects to be supported according to the second embodiment. It is a figure which shows the image.
  • FIG. 13 is a diagram showing information stored in the auxiliary storage device according to the present embodiment.
  • FIG. 14 is a diagram showing information stored in the auxiliary storage device according to the present embodiment.
  • FIG. 15 is a diagram showing information stored in the auxiliary storage device according to the present embodiment.
  • FIG. 16 is a diagram showing information stored in the auxiliary storage device according to the present embodiment.
  • 17 (a) to 17 (h) are diagrams showing a display pattern in a subject according to the second embodiment.
  • 18 (a) and 18 (b) are schematic views showing an example of a display of a user terminal according to the second embodiment.
  • a university window or a pharmacy window including a target person who is a student, a guardian of a student, a patient, etc., and a person who has a different position or role, such as a responder who is a worker who handles the target person.
  • the information on the object to be referred to by the responder will be explained.
  • the corresponding object is, for example, a document in the case of a university window and a drug in the case of a pharmacy window.
  • the workers are mainly the workers to perform the tasks.
  • Other workers for the work, customers who receive the work, etc. may be the target people, including those who have different positions and roles, and multiple people who share the work at the site.
  • the object is, for example, a device, a product, or a part installed at a work site or a place.
  • the information processing system 100 in this embodiment has an information processing device 1 as shown in FIG. 1, for example.
  • the information processing device 1 includes a central processing unit 2, a main storage device 3, and an auxiliary storage device 11.
  • the central processing unit 2 is, for example, a CPU (Central Processing Unit), and executes processing by calling a program stored in the main storage device 3.
  • the main storage device 3 is, for example, a RAM (RandomAccessMemory), which is an image acquisition unit 4, an image division unit 5, a scene estimation unit 6, a chunk estimation unit 7, a chunk output unit 8, and a first trained model, which will be described later.
  • a program such as a generation unit 9, a second learned model generation unit 10, a recommendation image output unit 13, and an object model identification unit 15 is stored.
  • a program including an image acquisition unit 4, an image division unit 5, a scene estimation unit 6, a chunk estimation unit 7, a chunk output unit 8, and a recommendation image output unit 13 may be called a control unit 15, and the first trained model generation may be performed.
  • a program including the unit 9 and the second trained model generation unit 10 may be referred to as a trained model generation unit 16.
  • the auxiliary storage device 11 is, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive), such as a first trained model DB 1, a first learning model DB 1', or a second trained model DB 2, which will be described later.
  • Databases such as the second learning model DB2', scene table TB1, model table TB2, content table TB3, scene content table TB4, content chunk table TB5, chunk metatable TB6, chunk table TB7, chunk metatable TB8, etc.
  • Stores tables such as the recommendation table TB9, the object table TB10, the object allocation table TB11, the annotation table TB12, the attention table TB13, the camera table TB14, and the roll table TB15.
  • the information processing apparatus 1 that outputs work information that is information about the work performed by the corresponding person has an image acquisition unit 4, an image division unit 5, a scene estimation unit 6, and chunk estimation at the usage stage.
  • a unit 7 and a chunk output unit 8 that outputs chunks that are information that divides or suggests work information are provided.
  • the work information may be referred to as content, and the content ID uniquely indicates the work information.
  • the image acquisition unit 4 is an original image 20 (FIG. 3) which is an image including a corresponding person 21 (FIG. 3) corresponding to the corresponding person and a plurality of corresponding objects 22 to 25 (FIG. 3) corresponding to the corresponding person.
  • a user terminal 12 such as a personal computer equipped with a camera.
  • the image segmentation unit 5 divides the original image 20 into an image of the person to be imaged by the person to be dealt with 21, an object identification information 61 for identifying the object (such as a person to be correspondent or a person to be correspondent), and each subject.
  • the corresponding objects 22 to 25 are divided into a plurality of imaged objects 40 to 43.
  • the scene estimation unit 6 estimates the scene that is the situation performed by the responder. Specifically, the scene estimation unit 6 acquires, for example, the corresponding person image 30 (35), for example, the corresponding person image, the target person identification information 61, and the like as the target person image. The scene estimation unit 6 uses, for example, the first trained model DB1 in which the association between the corresponding person image 30 (35) and the scene ID uniquely indicating the scene is stored. And estimate the scene. In addition to the corresponding person image 30 (35), the scene estimation unit 6 uses, for example, an image of the corresponding person (target person image), identification information for identifying the corresponding person and the corresponding person (target person identification information), and a scene. The scene may be estimated using the first trained model DB1 in which the association between the uniquely shown scene ID and the scene ID is stored.
  • the scene estimation unit 6 has a relationship between the corresponding person identification information and the scene ID uniquely indicating the scene in which the corresponding person performs the situation.
  • the first trained model in which is stored may be used to further estimate the scene.
  • the scene estimation unit 6 uniquely indicates the person identification information and the scene which is the situation performed by the person.
  • a first trained model in which the association between and is stored may be used to further estimate the scene.
  • the scene estimation unit 6 acquires a scene name using the scene ID as a search key from the scene table TB1 which is a table in which the scene ID and the scene name which is the name of the scene are linked one-to-one, and the user terminal 12 is used. Send.
  • the user terminal 12 presents the scene name received from the scene estimation unit 6 to the target person.
  • the presentation of the scene name is displayed, for example, on one side of the object model previously assigned by the target person by the display unit 17 described later.
  • the chunk estimation unit 7 acquires the corresponding object images 40 to 43, which are images of the corresponding object 22 (23 to 25) related to the work, and uniquely identifies the chunk with the corresponding object images 40 (41 to 43). Using one of a plurality of second trained model DB2s in which the association between the indicated chunk ID and one or more chunk meta-IDs associated one-to-one is stored. , Estimate chunks.
  • the chunk estimation unit 7 selects one of a plurality of second trained models in which the association between the plurality of object images, the chunk ID, and the plurality of chunk meta IDs is stored. For example, when the target person information is the corresponding person identification information, the chunk associated with the corresponding person identification information may be further estimated.
  • the chunk estimation unit 7 is one of a plurality of second trained models in which the association between the plurality of object images, the chunk ID, and the plurality of chunk meta IDs is stored. For example, when the target person information is the corresponding person identification information, the chunk associated with the corresponding person identification information may be further estimated.
  • the chunk estimation unit 7 selects one of the plurality of second trained model DB2s by using the model ID associated with the scene ID on a one-to-one basis. Further, the chunk meta ID uniquely indicates a chunk meta value which is information regarding the properties of the corresponding objects 22 to 25.
  • the chunk estimation unit 7 acquires the model ID from the model table TB2, which is a table in which the model ID and the scene ID are linked one-to-one, using the scene ID as a search key. Further, the chunk estimation unit 7 acquires the chunk ID from the chunk meta table TB6, which is a table in which the chunk ID and the chunk meta ID are linked one-to-one or one-to-many, using the chunk meta ID as a search key. ..
  • the chunk estimation unit 7 acquires a chunk summary showing the outline of the chunk from the chunk table TB7 using the chunk ID as a search key, and transmits the chunk summary to the user terminal 12.
  • the user terminal 12 presents the chunk summary received from the chunk estimation unit 7 to the target person.
  • the presentation of the chunk summary is displayed, for example, on one side of the object model previously assigned by the target person by the display unit 14 described later.
  • the chunk estimation unit 7 acquires chunks from the chunk table TB7 using the chunk ID as a search key, and transmits the chunks to the user terminal 12.
  • the user terminal 12 presents the chunk received from the chunk estimation unit 7 to the target person.
  • the chunk presentation is displayed, for example, on one side of the object model previously assigned by the target person by the display unit 14 described later.
  • the chunk table TB7 is a table in which chunks, chunk summaries, and hash values are associated with a chunk ID on a one-to-one basis.
  • the hash value is used, for example, to confirm whether or not the chunk has been changed.
  • the image acquisition unit 4 acquires the target person identification information for identifying the target person.
  • the target person identification information is, for example, a face image that identifies the responder and the respondent, and is, for example, a bar code of an ID card such as a photo certificate, a two-dimensional code, or the like. It may be imaged by a camera or the like.
  • the information processing apparatus 1 identifies the captured target person identification information and confirms that the responder is a legitimate target person (correspondent) for the work.
  • the target person identification information may, for example, identify a plurality of target persons, and may further identify a remote co-owner who shares information.
  • FIG. 13 shows the object model table TB10.
  • the object model table TB10 displays, for example, an object model ID for identifying the object model, operation information indicating the type of operation for the object model, a basic size in which the object model is displayed, and a plurality of object models 6 as information for identifying the object model. Possible additional number, display coordinates indicating the position where the object model is displayed, estimated scene ID and chunk ID, area ID (affiliation / department) that identifies the affiliation, department, location, etc. to perform work, target person (correspondence) Persons, respondents, etc.) and role IDs that identify the skills, attributes, roles, qualifications, etc. of the responders are stored in association with each other.
  • an object model ID for identifying the object model
  • operation information indicating the type of operation for the object model
  • a basic size in which the object model is displayed a plurality of object models 6 as information for identifying the object model.
  • Possible additional number display coordinates indicating the position where the object model is displayed, estimated scene ID
  • FIG. 14 shows the object allocation table TB11.
  • the object allocation table TB11 is, for example, linked to the object model ID, display area information regarding the number of display areas of the object model, display area ID to which the recommended image to be displayed and reference information are linked, scene ID, chunk ID. Etc. are stored in association with each.
  • FIG. 15 shows the annotation table TB12 and the attention table TB13.
  • the annotation table TB12 for example, a video ID for identifying a video related to work, a camera ID for identifying a camera that shot, a scene ID for identifying a work scene, a shooting time indicating the time when the video was shot, and a video were shot.
  • the shooting time, image quality information, image quality and viewpoint coordinates of the shot image, meta ID, attention ID, etc. are stored in association with each other.
  • the attention table TB13 is, for example, an attention ID that identifies the priority of chunks to be displayed, attention type information indicating the display of attention information, a scene ID, attention information that occupies the content, attention information data (content), and higher reference information. , High-ranking attention ID indicating the presence or absence, etc. are stored in association with each.
  • FIG. 16 shows the camera table TB14 and the roll table TB15.
  • the camera table TB14 can be operated, for example, a camera ID that identifies the camera to be photographed, an area ID in which the camera is installed, model information indicating camera specifications, operations, roles, etc., line-of-sight information, switching information, external connection information, and operation. Person ID, role ID, etc. are stored in association with each other.
  • the role table TB15 has a role ID, an employee ID, a name, a qualification ID, a department ID, an area ID, a related role ID, etc. that identify a target person (correspondent, respondent, other responder, co-owner, etc.). , Correspond to each and stored.
  • the information processing device 1 may further include a recommendation image output unit 13, and the auxiliary storage device 11 may further include a recommendation table TB9.
  • the recommendation image output unit 13 searches for a recommended object image using the recommendation table TB9 using a combination of the model ID and one or a plurality of chunk meta IDs as a search key.
  • the recommendation image output unit 13 outputs the searched recommended object image to the user terminal 12.
  • the recommended object to be image refers to an image of the object to which the original image 20 is not captured, but is presumed to be necessary.
  • the recommendation table TB9 is a table in which the combination of the model ID and the chunk meta ID and the recommended object image are linked one-to-one-to-one.
  • the recommendation image output unit 13 searches for one of the plurality of second trained models as a search key for a combination of a model ID and one or a plurality of chunk meta IDs associated with a scene ID on a one-to-one basis.
  • the shared person who shares information in the work and the shared information shared with the shared person may be further searched, and the recommended information associated with the shared person and the shared information may be output.
  • the recommendation image output unit is, for example, a collaborator who collaborates with the target person (correspondent), a trainer who is a leader of the target person (correspondent), or at least an inspector who monitors the target person (correspondent).
  • a person identified in any position may be searched as a shared person, and the recommended information associated with the shared person and the shared information may be output.
  • the display unit 14 allocates each of the chunks and the recommended image output by the chunk output unit 13 to each surface of the object model having a plurality of display areas, and assigns the assigned object model. It is displayed on the user terminal 12 via the user terminal 12.
  • the display unit 14 further includes an object model identification unit 15.
  • the object model specifying unit 15 identifies the object model that displays the recommended image and the recommended information output by the recommended image output unit 13, and associates the scene and chunk with the object model ID that uniquely indicates the object model. , Identify the object model.
  • the display unit 14 displays the recommendation image output by the recommendation image output unit 13 and the recommendation information in any of the display areas of the plurality of display areas included in the object model specified by the object model identification unit 15. , Assign as a state that can be shared with the shared person.
  • the object model specifying unit 15 uniquely displays an object model that displays reference information including at least a scene or chunk based on at least one of a scene estimated by the scene estimation unit 6 and a chunk estimated by the chunk estimation unit 7.
  • the object model is specified by associating with the indicated object model ID.
  • the object model specifying unit 15 refers to the object model table TB10 acquired by, for example, the image acquisition unit 4 and stored in the auxiliary storage device 11, and displays reference information narrowed down to the corresponding person when the corresponding person needs it. Identify a suitable object model 6 to do.
  • the object model specifying unit 15 refers to the object model table TB10 shown in FIG. 13, and based on, for example, an estimated scene ID, chunk ID, etc., based on various information such as a work area and a position of a corresponding person. Identify the object model that can be displayed under that condition.
  • the object model specifying unit 15 is aware of the information narrowed down to the target person and the corresponding person when the corresponding person is required by, for example, the basic size, display coordinates, area ID, etc. of the object model identified by the object model ID. Identify an object model that can present information to the responder that displays no useful information.
  • the object model specifying unit 15 identifies an object model, for example, a qualification ID for specifying skill information of a correspondent, display coordinates for specifying spatial information for performing work, a basic size for specifying feature information of an object, or work. It may be specified based on at least one of the work level information of.
  • the object model specifying unit 15 refers to the object model table based on the estimated scene ID, chunk ID, etc., and for example, from various data such as the shape, the basic size, and the number of additional objects associated with the object model ID, at least 2 It is also possible to specify an object model having a shape having a display area 8 equal to or larger than a surface, and capable of displaying a plurality of the same or different object models 6.
  • the object model specifying unit 15 refers to the object model table TB10 based on the estimated scene ID, chunk ID, etc., and for example, by various operation information for the object model 6, rotation display, enlarged display, based on the working state, You may want to specify an object model that can be displayed in at least one of reduced display, protruding display, vibration display, state display, discoloration display, and shading display.
  • the object model specifying unit 15 may be specified based on various information such as the positional relationship between the object and the corresponding person, the dominant hand, the language used, and the like, and the displayed position may be determined.
  • the object model specifying unit 15 identifies an object model having a display area that can be displayed based on, for example, the type of chunk estimated by the chunk estimation unit 7, the number of characters in the chunk, and the like, and the chunks are displayed in each display area of the object model. You may make an assignment.
  • the object model specifying unit 15 may preferentially display the customized object model.
  • the object model may be rotated or moved by grasping any of the left and right or upper and lower sides of the object model, for example, via the user terminal 12 worn by the corresponding person.
  • the object model specifying unit 15 projects the specified object model toward the corresponding person and displays it. May be good.
  • the display area of the object model specified by the object model specifying unit 15 includes at least a scene or chunk based on at least one of the scene estimated by the scene estimation unit 6 and the chunk estimated by the chunk estimation unit 7. Assign reference information.
  • the object model specifying unit 15 refers to the object allocation table TB11 shown in FIG. 14, for example, the estimated scene ID, the displayable information indicated by the chuck ID, the work area, the position of the correspondent, and the chunk data. Based on various information such as the amount, the object model specified by the object model specifying unit 15 is assigned to the display area.
  • the object model specifying unit 15 may refer to the attention table TB13 shown in FIG. 15 in addition to the object allocation table and allocate the chunks associated with the scene ID. good.
  • the attention ID of the attention table TB13 is referred to, and the presence or absence of the attention information set in association with the image of the object to be matched is determined.
  • the object model identification unit 15 has reference information (for example, a display having priority over the estimated chunk, or a display attached to the chunk). (Attention information, attention information data, etc.) are assigned.
  • reference information for example, a display having priority over the estimated chunk, or a display attached to the chunk.
  • the object model specifying unit 15 refers to the annotation table TB12 based on the estimated scene ID, chunk ID, etc., and for example, all or part of various videos and data associated with each of the attention IDs are the videos. May be assigned based on the time, length, and viewpoint at which the image was taken.
  • the position information of the room or place where the person in charge works for example, the position information of the room or place where the person in charge works, the environmental information such as the ambient temperature and humidity, the work of the target person, and the work of the target person.
  • the display of the display area in the object model may be assigned based on various information related to the movement, the biological information of the target person, and the like.
  • the object model specifying unit 15 has an object model, each display area of the object model, and a display area of the object model so that the virtual reality displayed on the transmissive display of the user terminal 12 is superimposed and displayed to the corresponding person. Set the recommended image to be assigned to the display area.
  • the object model specifying unit 15 may acquire evaluation target information including at least position information which is information on the position where a corresponding person is present and work-related information related to work.
  • the work-related information is, for example, information around the object to be addressed, the dominant arm that the responder (target person) works on, and the like, based on these, for example, "the work position of the responder is to the right of the object to be addressed", "The distance between the correspondent and the object to be dealt with is 3 meters", “there is no arrangement around the object to be correspondent", “the dominant arm of the correspondent is to the right", and the like may be displayed.
  • the object model specifying unit 15 determines that, for example, "the space on the left side of the object to be corresponded is empty", “the correspondent is right-handed”, and "the estimated chunk amount is 2 screens", the object model identification unit 15 is to be dealt with.
  • “Chunk (1B827-01.txt_0) ”and the like are set in the display area.
  • the object model specifying unit 15 is an object model specified based on, for example, a scene as a work place, a positional relationship between a correspondent and a corresponding object, chunk data or information amount as reference information, a size of a recommended image, and the like.
  • a plurality of cubes for example, a plurality of cubes may be positioned vertically or horizontally and displayed.
  • the object model specifying unit 15 may assign and display a working state or the like on the upper surface of the object model, for example, as a "face mark” indicating emotions and emotions of the face.
  • the face mark may be indicated by, for example, "normal (smile mark)", “abnormality (sadness mark)", or "notification (speech mark)".
  • the information processing device 1 in the learning stage will be described with reference to FIG.
  • the corresponding person image 30 (35) input from an input device (not shown) and one or a plurality of corresponding object images 40 to 43 are learned as a set.
  • learning refers to, for example, supervised learning.
  • the corresponding person image 30 (35) will be described as an example, but in addition to the corresponding person image 30 (35), for example, the corresponding person image, the corresponding person identification information, the corresponding person information, and the like may be used.
  • the target person including the responder and the subject, and the target person identification information 61 including the responder identification information and the respondent information.
  • FIG. 2 is a block diagram showing the configuration of the information processing apparatus 1 in the learning stage according to the present embodiment.
  • the information processing apparatus 1 includes a first trained model generation unit 9 and a second trained model generation unit 10.
  • the first trained model generation unit 9 generates the first trained model DB1 by training the first learning model DB1'with the scene ID and the corresponding person image 30 (35) as a pair. It is a program to do.
  • the first trained model generation unit 9 acquires the scene ID from the scene table TB1 with respect to the corresponding person image 30 (35), and acquires the model ID corresponding to the scene ID from the model table TB2.
  • the second trained model generation unit 10 designates a model ID and learns from the second learning model DB 2'with one or a plurality of chunk meta IDs and the corresponding object images 40 (41 to 43) as a pair. It is a program that generates the second trained model DB2 by making it.
  • the second learned model generation unit 10 acquires the content ID from the scene / content table TB4, which is a table in which the scene ID and the content ID are linked one-to-many, using the scene ID as a search key.
  • the scene ID that serves as a search key is associated with the corresponding person image 30 (35) that is paired with the corresponding object image 40 (41 to 43) to be processed.
  • the second learned model generation unit 10 acquires content from the content table TB3, which is a table in which the content ID and the content are linked one-to-one, using the content ID as a search key.
  • the second learned model generation unit 10 acquires the chunk ID from the content chunk table TB5, which is a table in which the content ID and the chunk ID are linked one-to-one or many, using the content ID as a search key.
  • the second learned model generation unit 10 acquires chunks from the chunk table TB7 using the chunk ID as a search key, and acquires a chunk meta ID from the chunk meta table TB6 using the chunk ID as a search key.
  • the second trained model generation unit 10 acquires the chunk meta value from the chunk meta table TB8 using the chunk meta ID as a search key.
  • the chunk meta table TB8 is a table in which the chunk category ID, the chunk category name, and the chunk meta value are linked to the chunk meta ID on a one-to-one basis.
  • the chunk category ID uniquely indicates the chunk category name, which is the name of the category to which the chunk meta value belongs.
  • the second trained model generation unit 10 refers to the corresponding object images 40 (41 to 43) and confirms that there is no problem in the acquired chunks, contents, and chunk meta values.
  • the second trained model generation unit 10 can generate a highly accurate trained model DB2, and the usage stage.
  • the information processing apparatus 1 can perform highly accurate processing.
  • FIG. 3 is a diagram showing an original image 20, a corresponding person image 30, a target person identification image 44, and a plurality of corresponding object images 40 to 43 according to the present embodiment.
  • the original image 20, the corresponding person image 30, the target person identification image 44, and the plurality of corresponding object images 40 to 43 are displayed on, for example, the user terminal 12.
  • FIG. 3 shows an example of being displayed at the same time, the original image 20, the corresponding person image 30, the target person identification image 44, and the plurality of corresponding object images 40 to 43 are separately displayed on the user terminal 12. You may.
  • the original image 20 captures the corresponding person 21, the target person identification image 44, and the corresponding objects 22 to 25.
  • the size of the corresponding objects 22 to 25 is estimated based on information that does not change for each scene in the booth such as a desk.
  • the corresponding objects 22 to 25 include, like the corresponding object 24, information on the contents such as the attached photo 26, the internal text 27, and the signature 28, as well as code information such as a bar code and a two-dimensional code.
  • code information such as a bar code and a two-dimensional code.
  • Various coupons and the like may be obtained.
  • the target person identification image 44 may include, for example, the target person identification information 61, the face photograph 61a of the target person (for example, the corresponding person or the person to be dealt with), the name of the target person 61b, and the barcode 61c for identifying the target person. good.
  • Code information such as barcodes and two-dimensional codes, and various coupons are printed on paper media in advance, or may be displayed on the screen of the user terminal 12 of the respondent or the respondent 21, for example. good.
  • FIG. 4 is a diagram showing a tree structure in which the respondent 21 and the plurality of counterparts 22 to 25 are related according to the present embodiment.
  • the corresponding person 21 for example, the corresponding person and the target person identification information (for example, the corresponding person identification information, the corresponding person identification information, etc.) may be used.
  • the subject will be described in detail by taking the respondent 21 as an example.
  • the image segmentation section 5 has a plurality of respondents 21 and a plurality of respondents 21 as a tree structure in which the respondent 21 is a root node and a plurality of counterparts 22 to 25 are leaf nodes or internal nodes. It is associated with the corresponding objects 22 to 25.
  • the image dividing unit 5 further includes information such as an attached photograph 26, a text 27, and a signature 28, which are information included in at least one of the objects 22 to 25, as well as code information such as a barcode and a two-dimensional code. , Various coupons, etc. may be acquired and associated with the tree structure as a leaf node.
  • FIG. 5 shows a first trained model DB1 and a second trained model DB2 according to the present embodiment.
  • the first trained model DB 1 includes a plurality of respondent images 30 (35), a plurality of scene IDs, and a plurality of respondent images 30 (35) generated by machine learning using a plurality of the trained model DB 1 as a pair of learning data. The connection between them is remembered.
  • machine learning is, for example, a convolutional neural network (CNN).
  • the association between the respondent image 30 (35) and the scene ID is specifically represented by the nodes indicated by circles in FIG. 5, the edges indicated by arrows, and the weighting factors set for the edges. It can be represented by a convolutional neural network. As shown in FIG. 5, the input of the corresponding person image 30 (35) to the first trained model DB1 is for each pixel such as pixels p1 and p2.
  • the second trained model DB2 is associated with the model ID on a one-to-one basis, and there are a plurality of them.
  • Each of the second trained model DB2 is generated by machine learning using a plurality of object images 40 (41 to 43) and one or a plurality of chunk meta IDs as a pair of training data.
  • the association between the plurality of object images 40 (41 to 43) and the plurality of one or a plurality of chunk meta IDs is stored.
  • machine learning is, for example, a convolutional neural network (CNN).
  • the association between the plurality of object images 40 (41 to 43) and the plurality of one or a plurality of chunk meta IDs is specifically the node indicated by a circle and the edge indicated by an arrow in FIG. It can be represented by a convolutional neural network represented by and the weighting factor set on the edge.
  • the input of the object image 40 (41 to 43) to the second trained model DB2 is for each pixel such as pixels p1 and p2.
  • FIG. 6 is a diagram showing information stored in the auxiliary storage device 11 according to the present embodiment.
  • the scene ID stored in the scene table TB1 or the like is a 3-digit hexadecimal number such as 0FD.
  • the scene name stored in the scene table TB1 or the like is, for example, a grade inquiry or a career counseling.
  • the model ID stored in the model table TB2 or the like is represented by a two-character alphabetic character and a one-digit decimal number, for example, MD1.
  • the content ID stored in the content table TB3 or the like is represented by a 5-digit hexadecimal number and a 2-digit decimal number, for example, 1B827-01.
  • the content stored in the content table TB3 or the like is, for example, 1B827-01.
  • a file name that is a content ID, such as txt, is indicated with an extension, and a pointer to the substance of the content is stored.
  • the chunk ID stored in the content chunk table TB5 or the like is represented by a 5-digit and 2-digit decimal number such as 82700-01.
  • the chunk meta ID stored in the chunk meta table TB6 or the like is a 4-digit hexadecimal number such as 24FD.
  • the chunks stored in the chunk table TB7 are, for example, 1B827-01. It is indicated by a file name of the content corresponding to the target chunk and a one-digit decimal number such as pxt_0, and a pointer to a part of the substance of the content corresponding to the target chunk is stored.
  • the chunk summary stored in the chunk table TB7 is a document summarizing the contents of the chunk, for example, "Hello Work, ".
  • the hash value stored in the chunk table TB7 is a 15-digit hexadecimal number such as 564544d8f0b746e.
  • the chunk category ID stored in the chunk meta table TB8 is a 3-digit decimal number such as 394.
  • the chunk category name stored in the chunk meta table TB8 is, for example, the size of the paper, the color of the paper, or the presence or absence of holes in the paper.
  • the chunk meta values stored in the chunk meta table TB8 are, for example, A4, B4, white, blue, with holes on the sides, and without holes.
  • the value of the chunk category ID and the chunk category name may be NULL.
  • the combination of chunk meta IDs stored in the recommendation table TB9 is (24FD, 83D9), (25FD), etc., and one or more chunk meta IDs are combined.
  • the recommended object image stored in the recommendation table TB9 is, for example, IMG001.
  • the shared object information stored in the recommendation table TB9 is, for example, IMG111.
  • Shared object information includes, for example, images, various video videos, texts such as documents and emails, online (online conferences, teleconferencing, videophones, etc.), applications for chats, link information, and the like. May be good.
  • the co-owner information stored in the recommendation table TB9 stores, for example, the attributes and conditions of the other party with whom the information is shared, such as the target person and the inspector, and the information for identifying the co-owner, in association with each other.
  • the information processing device 1 refers to, for example, the co-owner information stored in the recommendation table 7, and shares the shared object information through the display area of the object model described later.
  • the data structure of the work information has the chunk ID as the first layer and the chunk ID as the second layer. It has a hierarchical structure in which the content ID is a third layer and the scene ID is a fourth layer, which is the uppermost layer.
  • FIG. 7 is a sequence diagram for explaining the scene estimation function, chunk estimation function, and chunk output function according to the present embodiment.
  • the information processing functions at the usage stage include a scene estimation function realized by the scene estimation process S60 described later, a chunk estimation function realized by the chunk estimation process S80 described later, and a chunk output realized by the chunk output process S100 described later. It consists of functions.
  • the image acquisition unit 4 included in the control unit 15 receives the original image 20 from the user terminal 12 (S1).
  • the image segmentation unit 5 included in the control unit 15 divides the original image 20 into the corresponding person image 30 and the corresponding object images 40 to 43.
  • the image segmentation unit 5 transmits the corresponding person image 30 to the scene estimation unit 6 and transmits the corresponding object images 40 to 43 to the chunk estimation unit 7.
  • the scene estimation unit 6 included in the control unit 15 inputs the corresponding person image 30 into the first trained model DB 1 (S2).
  • the first trained model DB 1 selects one or a plurality of scene IDs strongly associated with the received image 30, and selects one or a plurality of scene IDs for the scene estimation unit 6 (hereinafter, this is the first first). It may be called a scene ID list) (S3).
  • the scene estimation unit 6 When the scene estimation unit 6 acquires the first scene ID list, it transmits it to the user terminal 12 as it is (S4).
  • the user terminal 12 transmits to the scene estimation unit 6 whether or not there is a cache for each scene ID included in the first scene ID list (S5).
  • the user terminal 12 holds a table equivalent to the scene table TB1 with respect to the information processed in the past.
  • the user terminal 12 searches the table held by the user terminal 12 using the scene ID of the received first scene ID list as a search key.
  • the scene ID in which the search result is found has a cache, and the scene ID in which the search result cannot be found has no cache.
  • the scene estimation unit 6 has one or a plurality of scene IDs (hereinafter, referred to as a second scene ID) having no cache in the user terminal 12 among the respective scene IDs included in the first scene ID list received from the user terminal 12.
  • the scene table TB1 is searched using (may be called a list) as a search key (S6).
  • the scene estimation unit 6 acquires, as a search result, a scene name corresponding to each scene ID included in the second scene ID list (hereinafter, this may be referred to as a scene name list) from the scene table TB1 (S7). ..
  • the scene estimation unit 6 transmits the acquired scene name list to the user terminal 12 as it is (S8).
  • the information processing apparatus 1 realizes a scene estimation function for estimating the scene of the person to be addressed image 30 by estimating the scene name in steps S1 to S8.
  • the user terminal 12 presents the received scene name list to the target person.
  • the presentation of the scene name list is displayed, for example, on one side of the object model previously assigned by the target person.
  • the subject selects, for example, one scene name from the presented scene name list.
  • the user terminal 12 transmits the scene name selected by the target person to the chunk estimation unit 7 included in the control unit 15 (S9).
  • the chunk estimation unit 7 uses the scene ID corresponding to the scene name received from the user terminal 12 as a search key (S10), searches the model table TB2, and acquires the model ID (S11).
  • the chunk estimation unit 7 receives the corresponding object image 40 (41 to 43) from the image segmentation unit 5 (S12).
  • the chunk estimation unit 7 designates one of the plurality of second trained model DB2s by the model ID acquired from the model table TB2, and designates the corresponding object images 40 (41 to 43) for the second learning. It is input to the completed model DB2 (S13).
  • the second trained model DB 2 selects one or a plurality of chunk meta IDs strongly associated with the object image 40 (41 to 43), and selects one or a plurality of the chunk estimation unit 7.
  • a plurality of one or a plurality of chunk meta IDs (hereinafter, this may be referred to as a chunk meta ID list) are output (S14).
  • the chunk estimation unit 7 searches the chunk meta table TB6 using each one or a plurality of chunk meta IDs included in the chunk meta ID list as a search key (S15).
  • the chunk estimation unit 7 acquires one or a plurality of chunk IDs (hereinafter, this may be referred to as a first chunk ID list) from the chunk metatable TB6 as a search result (S16).
  • the chunk estimation unit 7 transmits the acquired first chunk ID list to the user terminal 12 as it is (S17).
  • the user terminal 12 transmits to the chunk estimation unit 7 whether or not there is a cache for each chunk ID included in the first chunk ID list (S18).
  • the user terminal 12 holds a table including a chunk ID column and a chunk summary column in the chunk table TB7 with respect to the information processed in the past.
  • the user terminal 12 searches the table held by the user terminal 12 using the chunk ID of the received first chunk ID list as a search key. Chunk IDs for which search results are found are cached, and chunk IDs for which search results are not found are cached.
  • the chunk estimation unit 7 has one or a plurality of chunk IDs (hereinafter, referred to as a second chunk ID) having no cache in the user terminal 12 among the respective chunk IDs included in the first chunk ID list received from the user terminal 12.
  • the chunk table TB7 is searched using (which may be called a list) as a search key (S19).
  • the chunk estimation unit 7 acquires a chunk summary (hereinafter, this may be referred to as a chunk summary list) corresponding to each chunk ID included in the second chunk ID list as a search result from the chunk table TB7 (S20). ..
  • the chunk estimation unit 7 transmits the acquired chunk summary list to the user terminal 12 as it is (S21).
  • the information processing apparatus 1 realizes a chunk estimation function for estimating chunks of the corresponding object 22 (23 to 25) by estimating chunk summaries in steps S9 to S21.
  • the user terminal 12 presents the received chunk summary list to the target person.
  • the chunk summary list presentation is displayed, for example, on one side of the object model pre-assigned by the subject.
  • the subject selects, for example, one chunk summary from the presented chunk summary list.
  • the user terminal 12 transmits the chunk summary selected by the target person to the chunk output unit 8 included in the control unit 15 (S22).
  • the chunk output unit 8 uses the chunk ID corresponding to the chunk summary received from the user terminal 12 as a search key (S23), searches the chunk table TB7, and acquires chunks (S24).
  • the chunk output unit 8 transmits the acquired chunk to the user terminal 12 as it is (S25).
  • the user terminal 12 presents the received chunk to the user.
  • the chunk presentation is displayed, for example, on one side of the object model pre-assigned by the subject.
  • the information processing apparatus 1 realizes a chunk output function for outputting chunks of the corresponding object 22 (23 to 25) by steps S22 to S25.
  • FIG. 8 is a sequence diagram provided for explaining the first trained model generation function and the second trained model generation function according to the present embodiment.
  • the information processing functions at the learning stage include a first trained model generation function realized by the first trained model generation process and a second trained model generation function realized by the second trained model generation process. And consists of.
  • the first trained model generation unit 9 included in the trained model generation unit 16 is a set of a scene name to be processed, a corresponding person image 30, and one or a plurality of corresponding object images 40 to 43. Is determined, and the scene table TB1 generated in advance is searched for in the scene table TB1 using the scene name as a search key (S31).
  • the first trained model generation unit 9 acquires the scene ID from the scene table TB1 as a search result (S32), and sets the corresponding person image 30 and the scene ID in the first learning model DB 1'as a pair. (S33).
  • the first trained model generation unit 9 transmits the acquired scene ID to the model table TB2 and makes a model ID acquisition request (S34).
  • the model table TB2 generates a model ID corresponding to the received scene ID and stores the combination of the scene ID and the model ID.
  • the first trained model generation unit 9 acquires the model ID from the model table TB2 (S35).
  • the information processing apparatus 1 realizes the first trained model generation function of generating the first trained model DB1 by steps S31 to S35.
  • the second trained model generation unit 10 included in the trained model generation unit 16 uses the scene ID received by the first trained model generation unit 9 in step S32 as a search key to generate a scene content table in advance. Search for TB4 (S36).
  • the second learned model generation unit 10 acquires the content ID from the scene content table TB4 as a search result (S37), and searches the content table TB3 generated in advance using the acquired content ID as a search key (S38). ..
  • the second learned model generation unit 10 acquires the content from the content table TB3 as a search result (S39), and searches the content chunk table TB5 generated in advance using the content ID acquired in step S37 as a search key (S). S40).
  • the second trained model generation unit 10 acquires the chunk ID from the content chunk table TB5 as a search result (S41), and searches the chunk table TB7 generated in advance using the acquired chunk ID as a search key (S42). ..
  • the second trained model generation unit 10 acquires chunks from the chunk table TB7 as a search result (S43), and searches the chunk metatable TB6 generated in advance using the chunk ID acquired in step S41 as a search key (S). S44).
  • the second trained model generation unit 10 acquires one or a plurality of chunk meta IDs from the chunk meta table TB6 as search results (S45), and pre-generates each acquired meta ID for chunks as a search key.
  • the meta table TB8 for chunks is searched (S46).
  • the second trained model generation unit 10 acquires the chunk meta value corresponding to each chunk meta ID as a search result from the chunk meta table TB8 (S47).
  • the second trained model generation unit 10 checks whether there is a problem in the content acquired in step S39, the chunk acquired in step S43, and the respective chunk meta values acquired in step S47, with the corresponding person image 30 and the corresponding. Confirmation is performed with reference to the object images 40 to 43.
  • the second trained model generation unit 10 confirms by referring to the facial expressions of the person to be corresponded 21 and the document names described in the objects to be dealt with 22 to 25.
  • the second trained model generation unit 10 determines, for example, the facial expression of the corresponding person 21 from the corresponding person image 30, and assigns the document names described in the corresponding objects 22 to 25 from the corresponding object images 40 to 43. judge.
  • the second trained model generation unit 10 sets a pair of the model ID, the corresponding object images 40 (41 to 43), and one or a plurality of chunk meta IDs in the second learning model DB 2'. (S48).
  • the information processing apparatus 1 realizes the second trained model generation function of generating the second trained model DB 2 by steps S36 to S48.
  • FIG. 9 is a flowchart showing a processing procedure of information processing in the usage stage according to the present embodiment.
  • Information processing in the usage stage is composed of a scene estimation process S60, a chunk estimation process S80, and a chunk output process S100.
  • the scene estimation process S60 is composed of steps S61 to S67.
  • the scene estimation unit 6 receives the corresponding person image 30 (35) from the image segmentation unit 5 (S61)
  • the scene estimation unit 6 inputs the corresponding person image 30 (35) into the first trained model DB 1 (S62).
  • the scene estimation unit 6 acquires the first scene ID list as output from the first trained model DB 1 (S63), transmits the first scene ID list to the user terminal 12 as it is, and determines whether or not there is a cache. Contact 12 (S64).
  • the scene estimation process S60 ends and the chunk estimation process S80 starts.
  • the scene estimation unit 6 acquires the scene name list from the scene table TB1 (S66) and transmits it to the user terminal 12 as it is (S67). ), The scene estimation process S60 ends.
  • the chunk estimation process S80 is composed of steps S81 to S88.
  • the chunk estimation unit 7 receives the scene name selected by the target person from the user terminal 12 (S81).
  • the chunk estimation unit 7 Upon receiving the scene name from the user terminal 12, the chunk estimation unit 7 acquires the model ID from the model table TB2 (S82). Next, the chunk estimation unit 7 designates one of the plurality of second learned models DB 2 by the model ID, and designates the corresponding object images 40 (41 to 43) received from the image division unit 5. It is input to the trained model DB2 of 2 (S83).
  • the chunk estimation unit 7 acquires a chunk meta ID list as an output from the second trained model DB 2 (S84), and acquires a first chunk ID list from the chunk meta table TB6 (S85). Next, the chunk estimation unit 7 transmits the first chunk ID list to the user terminal 12 as it is, and inquires the user terminal 12 whether or not there is a cache (S86).
  • the chunk estimation process S80 ends and the chunk output process S100 starts.
  • the chunk estimation unit 7 acquires a chunk summary list from the chunk table TB7 (S87) and transmits it to the user terminal 12 as it is (S88). ), The chunk estimation process S80 ends.
  • the chunk output process S100 is composed of steps S101 to S103.
  • the chunk output unit 8 receives the chunk summary selected by the target person from the user terminal 12 (S101).
  • the chunk output unit 8 acquires the chunk from the chunk table TB7 (S102) and transmits it to the user terminal 12 as it is (S103), and the chunk output process S100 ends.
  • FIG. 10 is a flowchart showing information processing in the display stage according to the present embodiment.
  • the display on the display unit 14 is composed of the display process S110.
  • the display process S110 is composed of steps S111 to S114.
  • the display unit 14 includes an object model specifying unit 15, for example, acquiring object model information stored in advance (S111) and acquiring information on each display area constituting the object model (S112). Specifically, for the information of each display area, for example, the object model specifying unit 15 specifies the recommended image output by the recommended image output unit and the object model for displaying the recommended information.
  • the object model specifying unit 15 identifies the object model by associating the scene and chunk with the object model ID that uniquely indicates the object model.
  • the object model is specified, for example, with reference to the object model table TB10, and is specified based on various information such as a scene ID, a chunk ID, an area ID such as a target person or an information sharer, or a role ID. You may.
  • the display unit 14 is not assigned a display area of the object model based on various information such as, for example, the attribute, type, amount of information of the information presented to the user terminal, the type of the target person, the operation of the target person, and the like.
  • S113: NO displays information other than the display target and default display.
  • various information to be displayed is assigned to each display area and displayed (S114).
  • Allocation of various information (objects) to be displayed to each display area of the object model refers to the object allocation table TB11, and is assigned to the display area based on, for example, a scene ID, a chunk ID, a role ID, and the like. ..
  • the display area is allocated based on, for example, the display area information identified by the object model ID and the display area ID.
  • the display unit 14 displays the recommendation image and the recommendation information output by the recommendation image output unit in the display area of any of the plurality of display areas included in the object model specified by the object model identification unit 15 with the target person. It is assigned and displayed as a state that can be shared with the co-owner, and the display process S110 ends.
  • FIG. 11 is a flowchart showing a processing procedure of information processing in the learning stage according to the present embodiment.
  • Information processing in the learning stage is composed of a first trained model generation process S120 and a second trained model generation process S140.
  • the first trained model generation process S120 is composed of steps S121 to S124.
  • the first trained model generation unit 9 determines a set of the scene name, the corresponding person image 30 (35), and one or more corresponding object images 40 (41 to 43), the scene name is determined. Search the scene table TB1 as a search key (S121).
  • the first learned model generation unit 9 acquires the scene ID from the scene table TB1 as a search result (S122), and puts the scene ID and the corresponding person image 30 (35) in the first learning model DB 1'. Learn as a pair (S123).
  • the first trained model generation unit 9 transmits the scene ID acquired in step S122 to the model table TB2, makes a model ID acquisition request, and acquires the model ID (S124).
  • the second trained model generation process S140 is composed of steps S141 to S150.
  • the second learned model generation unit 10 searches the scene content table TB4 using the scene ID acquired in step S122 as a search key, and acquires the content ID (S141).
  • the second learned model generation unit 10 searches the content table TB3 using the acquired content ID as a search key and acquires the content (S142). Further, the second learned model generation unit 10 searches the content chunk table TB5 using the acquired content ID as a search key, and acquires the chunk ID (S143).
  • the second trained model generation unit 10 searches the chunk table TB7 using the acquired chunk ID as a search key and acquires chunks (S144). Further, the second learned model generation unit 10 searches the chunk meta table TB6 using the acquired chunk ID as a search key, and acquires one or a plurality of chunk meta IDs (S145).
  • the second trained model generation unit 10 searches the chunk meta table TB8 using each of the acquired one or a plurality of chunk meta IDs as a search key, and the chunk meta value corresponding to each chunk meta ID. (S146).
  • the second trained model generation unit 10 checks whether there is a problem with the content acquired in step S142, the chunk acquired in step S144, and the respective chunk meta values acquired in step S146 (corresponding person image 30 (35)). And the corresponding object images 40 (41 to 43) are referred to for confirmation (S147).
  • the second trained model generation unit 10 has a model ID, one or more chunk meta IDs, and a corresponding object image in the second learning model DB 2'. 40 (41 to 43) and 40 (41 to 43) are learned as a pair (S149), and the information processing in the learning stage regarding the set being processed is completed.
  • the chunk that divides or suggests the work information by the information processing apparatus 1 according to the present embodiment is presented via the user terminal 12. Therefore, it is possible to present the required amount of information by appropriately setting chunks. Also, if the chunk is information that suggests the entire document, there is no need to reconstruct the information on a large scale.
  • FIG. 12A is a schematic diagram showing an example of the operation of the information processing system 100
  • FIG. 12B is a diagram showing an original image, a target person image, and a plurality of objects to be imaged in the information processing system 100.
  • Is. 17 (a) to 17 (h) are diagrams showing display patterns in the target person in the information processing system 100
  • FIGS. 18 (a) to 18 (b) are user terminals 12 in the information processing system 100.
  • It is a schematic diagram which shows an example of the display of.
  • the target person (correspondent, correspondent, co-owner, trainee, etc.) 50a with respect to the corresponding object 60 in the manufacturing area.
  • the case where the work information about the work is output when the work is performed is shown.
  • the information processing system 100 is, for example, via a user terminal 12 (for example, a head-mounted display) worn by the target person 50a, for example, an employee ID card 61 which is the target person identification information, and a corresponding object 60 which performs work. Get an image.
  • the target person identification information may be, for example, an image of the subject's face, fingerprint, palmistry, vein, or the like, and may be unique information that can identify the target person.
  • the information processing system 100 acquires the target person identification information of the employee ID card 61 taken by the target person 50b if, for example, the target person 50b who collaborates with the target person 50b is in the same work area in addition to the target person 50a in the manufacturing area. You may try to do it.
  • the information processing system 100 acquires, for example, the target person identification information of the employee ID card 61 of the target person 50b and the image of the corresponding object 60 captured from the target person 50b side from the target person 50b.
  • the information processing system 100 for example, when a plurality of target persons 50a, target persons 50b, and the like collaborate in a manufacturing area, for example, an image taken by a camera of each user device 12 is used as a corresponding object in the collaborative work. It may be specified as an image of the image or divided.
  • the information processing system 100 may, for example, determine the divided image and search for the recommended recommended object image and the shared shared object information.
  • the information processing system 100 needs to share information in the work of the target person, for example, an image of a corresponding object that is not captured in the original image but is presumed to be necessary in the original image. It is possible to output a recommended image including information on the presumed object to be handled and the work of the target person, and it is possible to prevent forgetting to work.
  • the manufacturing area is connected to the customer / support area, which is the area of the customer / trainer 51, via a communication network such as the Internet, and the monitoring is, for example, the area of the inspector 52 in which the target person 50a monitors the work. Connected to the area.
  • the target person 50a outputs information necessary for one work from a viewpoint according to each position to the customer / trainer 51 and the inspector 52 from various aspects, and has a plurality of different positions, work places, and work hours. Enables information sharing among workers.
  • the target person 50a shares information in the customer / trainer 51 and the inspector 52 via the above-mentioned object model.
  • the information processing system 100 searches for, for example, a recommended recommended object image and shared shared object information by the recommendation image output unit 13, and although it is not captured in the original image, it is originally necessary. It outputs a recommended image including an image of the object to be presumed to be, an object to be presumed to need information sharing in the work of the subject, and information on the work of the subject.
  • the recommendation image output unit 13 allocates and outputs to a plurality of display areas of the object model.
  • FIG. 12B is a diagram showing an original image, a target person image, and a plurality of objects to be imaged in the information processing system 100.
  • the information processing system 100 acquires, for example, the target person identification information 70 regarding the target person and the corresponding object 60 image taken by the user terminal 12 worn by the target person 50a by the image acquisition unit 4, and causes the auxiliary storage device 11 to acquire the target person identification information 70 and the corresponding object 60 image. It is associated with each and stored.
  • the image stored in the auxiliary storage device 11 may be, for example, an image of a target person or target person identification information, for example, an image of an employee ID card 61.
  • the employee ID card 61 may include, for example, a face image 61a, a name 61b, and code information 61c of a person who performs work.
  • the image stored in the auxiliary storage device 11 includes, for example, an image of the corresponding object 60.
  • the image of the object 60 may include, for example, images of the parts 60a, 60b, and 60c constituting the object 60.
  • the original image captured by the image acquisition unit 4 is divided by the image segmentation unit 5.
  • the image segmentation unit 4 may be divided into parts 60a to 60c, for example, after the image acquisition unit 4 acquires an image of the object 60.
  • the target person identification information may be, for example, information for identifying the target person, and may be, for example, an image of the employee ID card 61.
  • the target person identification information is, for example, an employee ID card 61, the face image 61a, the name 61b, and the code information 61c of the target person who performs the work may be included.
  • the images divided by the original image segmentation unit 4 acquired by the image acquisition unit 4 are stored as images 70, 71, 71a to 71c in association with each other in the auxiliary storage device 11.
  • FIG. 17 shows various display contents assigned and displayed by the object model specifying unit 15 to a plurality of display areas of the object model specified by the object model specifying unit 15 in the user terminal 2.
  • FIG. 17A is an example in which a plurality of scene candidates estimated by the scene estimation unit 6 are displayed in one display area of the object model.
  • FIG. 17B is an example in which content / difference information is displayed in the display area as, for example, an image or information associated with a chunk ID.
  • the user terminal 12 is a device such as a smartphone, the image information of the user's viewpoint is switched to the rear camera, and the object to be photographed is photographed as a second image, and the photographed object is displayed in the display area. It is an example displayed in.
  • FIG. 17C for example, the user terminal 12 is a device such as a smartphone, the image information of the user's viewpoint is switched to the rear camera, and the object to be photographed is photographed as a second image, and the photographed object is displayed in the display area. It is an example displayed in.
  • FIG. 17C for example, the user terminal 12 is a device such as a smartphone, the image information of the user's viewpoint is switched to the rear camera, and the object to be photographed is photographed
  • FIG. 17D for example, a skilled person (trainer) photographs a target person (trainee) to work on a corresponding object, and a work checklist displayed based on the second image is displayed in the display area. It is an example to be done.
  • FIG. 17E is an example of being displayed in the display area as an image of a corresponding object recorded by, for example, a user terminal end 12 of a skilled person (trainer).
  • FIG. 17 (f) is an example in which, for example, an image for confirming the behavior of a target person performing a work from a bird's-eye view and related information are displayed together in a display area.
  • FIG. 17 (g) is an example in which the work information for generating the expert / AI learning data when acquiring the learning information recorded by the expert (trainer) is displayed in the display area.
  • FIG. 17H is an example in which related moving image information and origin information are displayed in the display area as related narrative information associated with the chunk ID, for example.
  • each reference information displayed in FIGS. 17A to 17H is displayed as a recommended image.
  • the scene estimation unit 6 selects a scene and estimates chunks.
  • FIG. 17A may be displayed in front of the subject.
  • the display area of the object model may be rotated based on the work content, work status, etc. of the target person, and more important information, attention information, and the like may be preferentially displayed.
  • FIGS. 18A to 18B show the display contents on the user terminal 12 by the information processing system 100.
  • the user terminal 12 is a smartphone and is displayed as a flat display.
  • chunks and recommendations are made in the display areas 80a of the display screen 80 of the target person (correspondent) such as a smartphone or tablet, and in the respective display areas of the specified object models 80b and 80c. This is an example in which various information such as images are displayed.
  • the image of the object model may be shared between different user terminals 12.
  • the image of the object model may be shared between different user terminals 12.
  • the user terminal 12 is a personal computer or the like, and the target person is an inspector.
  • a display area 81a for displaying a bird's-eye view of the target person working on the object to be handled
  • a display area 81b for displaying a viewpoint image of a skilled person in charge of the skilled person
  • the target person for example, a trainer, a trainee, etc.
  • a display area 81c for selecting an object model for transmitting information and a display area 81d for transmitting an alert to a target person for example, a trainer, a trainee, etc.
  • the target person needs it, the necessary information, shared information, related information, etc. are presented to the target person (correspondent), etc. from a viewpoint and role different from the information narrowed down to the target person. Is possible.
  • the effectiveness of information provision and the usefulness of information can be further improved.
  • the object model specifying unit 15 may allocate, for example, schematic information to the display area.
  • Schematized information includes, for example, figures and illustrations showing facial expressions such as “smile”, “anxiety”, and “tension” that simplify human facial expressions, and “caution” and “warning” for the work situation of the subject.
  • Words and messages, as well as light emission states such as red, blue, and yellow may be displayed as indicator lights and the like.
  • model table TB2 By using the model table TB2, even if the relationship between the first trained model DB1 and the second trained model DB2 changes, it is possible to deal with it simply by changing the model table TB2, which is excellent in maintainability. Can provide the equipment.
  • the image acquisition unit 4, the image division unit 5, the scene estimation unit 6, the chunk estimation unit 7, the chunk output unit 8, the first trained model generation unit 9, and the second trained model generation unit 9 are used.
  • the 10 and the recommendation image output unit 13 are programmed, but the program is not limited to this, and a logic circuit may be used.
  • the table TB15 is not mounted on one device, but may be distributed and mounted on a plurality of devices connected by a network.
  • the present invention is not limited to this, and the first method is not limited to this.
  • the trained model DB1 and the second trained model DB2 may be generated separately.
  • the first trained model DB1 and the second trained model DB2 are generated separately, for example, when the scene is an existing one and only the content is added, it is not necessary to learn about the scene.
  • the present invention is not limited to this, and only one second trained model DB2 may be used.
  • the case of displaying the image of the corresponding object that is presumed to be originally necessary has been described, but the present invention is not limited to this, and a part of the corresponding object that is presumed to be originally necessary is displayed. May be displayed. Further, in the present embodiment, a corresponding object or a part of the corresponding object, which is presumed to be originally unnecessary, may be suggested.
  • the information processing apparatus 1 of the present embodiment is composed of a tree structure associated with the image dividing unit and values output from the first trained model DB1 and the second trained model DB2 in the usage stage. By comparing with the hierarchical structure, excess / deficiency points may be determined.
  • the recommendation image and the recommendation information assigned to the plurality of display areas of the object model by the display unit 14 are linked to the scene information and the scene information indicating the contents of the scene estimated by the scene estimation unit 6.
  • Work information related to the work performed by the target person work check information indicating the work process related to the work performed by the target person linked to the work information, chunk information linked to the work information, difference information of the work content related to the work, hand by the work trainer Includes at least one of model information showing the content of this work, work information showing the work video of the work scene of the target person, or instruction information shown according to the difference in work between the model information and the work information. You may do so.
  • the display unit 14 may output the object model in the vicinity of the corresponding object in the virtual display space of the target person. Further, the display unit 14 may output the object model by fixing the display position in the vicinity of the corresponding object in the virtual display space of the target person.
  • the object model specifying unit specifies the object model based on at least one of the skill information of the target person, the spatial information for performing the work, the characteristic information of the corresponding object, or the work level information for the work. You may do so.
  • the object model specifying unit may specify one or more object models having two or more display areas.
  • the object model identification unit is in at least one of rotation display, enlarged display, reduced display, protrusion display, vibration display, state display, discoloration display, and shading display based on the state of the work performed by the subject.
  • the object model may be displayed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Physics & Mathematics (AREA)
  • Economics (AREA)
  • Strategic Management (AREA)
  • Databases & Information Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Marketing (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Operations Research (AREA)
  • General Business, Economics & Management (AREA)
  • Game Theory and Decision Science (AREA)
  • Development Economics (AREA)
  • Computational Linguistics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

Le problème décrit par la présente invention est de fournir un appareil de traitement d'informations, un procédé de traitement d'informations et un système de traitement d'informations qui permettent, lorsque cela est requis par un opérateur, de fournir une quantité nécessaire d'informations à l'opérateur et à un co-opérateur, sans nécessiter une reconstruction d'informations à grande échelle. La solution selon l'invention porte sur un appareil de traitement d'informations qui est destiné à délivrer des informations de travail relatives à un travail à effectuer par un opérateur, et qui comporte : une unité d'acquisition d'image qui acquiert des images originales qui comprennent des personnes sujet comprenant un opérateur et une personne recevant une opération et comprennent une pluralité d'éléments d'opération à traiter par l'opérateur ; une unité de division d'image qui divise les images originales en une image de personne sujet dans laquelle les personnes sujets sont capturées et une pluralité d'images d'élément d'opération dans lesquelles les éléments d'opération sont capturés ; une unité de déduction de scène qui déduit une scène à l'aide d'un premier modèle formé ; une unité de déduction de fragment qui déduit un fragment en utilisant un modèle d'une pluralité de seconds modèles formés ; une unité de sortie qui délivre le fragment ; une unité de sortie d'image de recommandation qui recherche une image d'élément d'opération de recommandation et délivre une image de recommandation ; et une unité d'affichage qui attribue le fragment émis et l'image de recommandation émise à une zone d'affichage de modèle d'objet et l'affiche. L'unité de déduction de fragment sélectionne un modèle de la pluralité des seconds modèles formés à l'aide d'un identifiant de modèle associé à un ID de scène sur une base individuelle. Un méta-ID pour un fragment indique de manière unique une méta-valeur correspondant au fragment, qui représente des informations relatives à la propriété d'un élément d'opération.
PCT/JP2021/045713 2020-12-11 2021-12-10 Appareil de traitement d'informations, procédé de traitement d'informations et système de traitement d'informations WO2022124419A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2022568365A JPWO2022124419A1 (fr) 2020-12-11 2021-12-10

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020205948 2020-12-11
JP2020-205948 2020-12-11

Publications (1)

Publication Number Publication Date
WO2022124419A1 true WO2022124419A1 (fr) 2022-06-16

Family

ID=81974603

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/045713 WO2022124419A1 (fr) 2020-12-11 2021-12-10 Appareil de traitement d'informations, procédé de traitement d'informations et système de traitement d'informations

Country Status (2)

Country Link
JP (1) JPWO2022124419A1 (fr)
WO (1) WO2022124419A1 (fr)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005216137A (ja) * 2004-01-30 2005-08-11 Chugoku Electric Power Co Inc:The 保守支援システムおよび方法
JP2009529736A (ja) * 2006-03-10 2009-08-20 ネロ アーゲー 映像フレームのシーケンスを提供するための装置および方法、シーンモデルを提供するための装置および方法、シーンモデル、メニュー構造を作成するための装置および方法およびコンピュータ・プログラム
JP2012150613A (ja) * 2011-01-18 2012-08-09 Ricoh Co Ltd 作業内容測定装置及び作業管理装置
JP6607590B1 (ja) * 2019-03-29 2019-11-20 株式会社 情報システムエンジニアリング 情報提供システム及び情報提供方法
WO2020145085A1 (fr) * 2019-01-08 2020-07-16 株式会社日立国際電気 Dispositif de reconnaissance d'image, programme de reconnaissance d'image et procédé de reconnaissance d'image
JP2020528626A (ja) * 2017-07-27 2020-09-24 ベステル エレクトロニク サナイー ベ ティカレト エー．エス． ウェブページを三次元オブジェクト上に重ね合わせる方法、装置およびコンピュータプログラム
JP2020166353A (ja) * 2019-03-28 2020-10-08 Kddi株式会社 ロボット制御装置、ロボット制御方法及びロボット

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005216137A (ja) * 2004-01-30 2005-08-11 Chugoku Electric Power Co Inc:The 保守支援システムおよび方法
JP2009529736A (ja) * 2006-03-10 2009-08-20 ネロ アーゲー 映像フレームのシーケンスを提供するための装置および方法、シーンモデルを提供するための装置および方法、シーンモデル、メニュー構造を作成するための装置および方法およびコンピュータ・プログラム
JP2012150613A (ja) * 2011-01-18 2012-08-09 Ricoh Co Ltd 作業内容測定装置及び作業管理装置
JP2020528626A (ja) * 2017-07-27 2020-09-24 ベステル エレクトロニク サナイー ベ ティカレト エー．エス． ウェブページを三次元オブジェクト上に重ね合わせる方法、装置およびコンピュータプログラム
WO2020145085A1 (fr) * 2019-01-08 2020-07-16 株式会社日立国際電気 Dispositif de reconnaissance d'image, programme de reconnaissance d'image et procédé de reconnaissance d'image
JP2020166353A (ja) * 2019-03-28 2020-10-08 Kddi株式会社 ロボット制御装置、ロボット制御方法及びロボット
JP6607590B1 (ja) * 2019-03-29 2019-11-20 株式会社 情報システムエンジニアリング 情報提供システム及び情報提供方法

Also Published As

Publication number Publication date
JPWO2022124419A1 (fr) 2022-06-16

Similar Documents

Publication Publication Date Title
US10691876B2 (en) Networking in a social network
US9524418B2 (en) Systems and methods for detecting, identifying and tracking objects and events over time
Stevens et al. Seeing infrastructure: Race, facial recognition and the politics of data
JP2015531107A (ja) ソーシャルネットワーキングシステムを使用するイベントへの追加のユーザの推薦
US20170032298A1 (en) Methods and systems for visualizing individual and group skill profiles
US20200050853A1 (en) Method and system for leveraging ocr and machine learning to uncover reuse opportunities from collaboration boards
US11900301B2 (en) Information processing device and information processing method
JP2018181257A (ja) 面接管理プログラム及び面接管理装置
WO2022124419A1 (fr) Appareil de traitement d'informations, procédé de traitement d'informations et système de traitement d'informations
JP6124354B2 (ja) 経験学習支援システム、経験学習支援方法、情報処理装置およびその制御方法と制御プログラム
US9811893B2 (en) Composable situational awareness visualization system
JP2021174502A (ja) 情報処理システム、情報処理装置、情報処理方法、情報処理プログラム、通信端末およびその制御方法と制御プログラム
JP7340806B2 (ja) 画像検査装置、画像検査システム、および画像検査用プログラム
US20220394098A1 (en) Information processing system, system, and information processing method
JP2022136068A (ja) 情報表示装置、情報表示システム、情報表示プログラム、学習方法及びデータ構造
JP6324284B2 (ja) グループ学習システム
KR102349974B1 (ko) 학원 홍보물 제공 서비스 제공방법
JP2008020939A (ja) 原稿処理を支援する画像処理装置、原稿処理支援方法、およびコンピュータプログラム
KR100445688B1 (ko) 화상 채팅 서비스를 이용한 광고 제공 시스템 및 방법
JP2021163119A (ja) セミナー運営システム、セミナー運営方法、及びサーバ装置
JP2021108087A (ja) 管理システム
KR20200130552A (ko) 직업영상 공유 시스템 및 방법
CN109344249A (zh) 信息处理方法、装置、电子设备及计算机可读存储介质
WO2021193136A1 (fr) Dispositif de traitement d'informations et procédé de traitement d'informations
JP6818308B1 (ja) 情報処理装置及び情報処理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21903515

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022568365

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21903515

Country of ref document: EP

Kind code of ref document: A1