JP2022139822A - Display control apparatus, display control program, and display control method - Google Patents

Abstract

To provide a technique which allows easier selection of content relating to a work.SOLUTION: The present invention is directed to a display control apparatus 10 according to an embodiment for displaying content at a work site where a work is subjected to a predetermined processing by a user. The apparatus has an image recognition unit 101 for recognizing the work based on a captured image obtained by capturing the work site, an information management unit 103 for managing content information associating a kind of the work with a kind of the work recognized based on the content information, a content selection unit 104 for selecting the content associated with the recognized work based on the content information, and a display control unit 105 for displaying, in a display device, the selected content at the work site.SELECTED DRAWING: Figure 3

Description

TECHNICAL FIELD Embodiments of the present invention relate to techniques for displaying content.

Conventionally, at facilities where workers are engaged, especially at work sites where predetermined processing is performed on works, workers have been able to view content including work details using tablet terminals/head-mounted displays. there is

In addition, as related technologies, design shape data storage means for storing design shape data of the construction surface to be constructed, current shape data acquisition means for acquiring current shape data of the construction surface formed in the construction process, design shape Based on the comparison with the data and the current shape data, construction state management information creating means for creating construction state management information consisting of the degree of matching between the design shape data and the current shape data, so that the worker can be identified, A construction management apparatus is known that includes construction state management information projecting means for projecting construction management information onto a construction surface in a construction process (Patent Document 1).

JP 2017-117146 A

Depending on the work site, the period during which a worker performs the same type of processing on the same type of work is short, and when engaging in a new type of work, the worker confirms the work content by visually recognizing the content each time. It will be done. At this time, the worker has to search for the content related to the work assigned to him/herself from a plurality of contents, and there is a problem that the selection operation of the content is troublesome.

The embodiments of the present invention have been made to solve the above-described problems, and an object thereof is to provide a technology that enables easier selection of work-related content.

In order to solve the above-described problems, an embodiment of the present invention is a display control device that displays content at a work site where a user performs predetermined processing on a work, the display control device comprising: an image recognition unit that recognizes a work; an information management unit that manages content information in which a type of work and content related to the work belonging to the type are associated; and a type of work recognized based on the content information. A content selection unit that selects associated content, and a display control unit that displays the selected content on a display device at the work site.

1 is a schematic diagram showing the configuration of a display control system according to an embodiment; FIG. It is a block diagram which shows the hardware constitutions of the display control apparatus which concerns on embodiment. 1 is a block diagram showing the functional configuration of a display control device according to an embodiment; FIG. 4 is a flowchart showing the overall operation of the display control device according to the embodiment; 4 is a flowchart showing the operation of image recognition processing; 4 is a flow chart showing the operation of content selection processing; 4 is a flow chart showing the operation of display control processing. It is a schematic diagram showing a setting range. FIG. 3 is a schematic diagram showing displayed content;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Display control system)
The configuration of the display control system according to this embodiment and the work site where this display control system is operated will be described. FIG. 1 is a schematic diagram showing the configuration of a display control system according to this embodiment.

As shown in FIG. 1, the display control system according to this embodiment includes a display control device 10, a projection device 20, an imaging device 30, a head mounted display 40, and a terminal device 50. In addition, in this embodiment, the display control system shall be provided with two projection apparatuses 20 and one imaging device 30 for description. The display control device 10 is communicably connected to each of the projection device 20, the imaging device 30, the head mounted display 40, and the terminal device 50. FIG.

The display control system is operated at a work site where a user U as an operator performs predetermined processing on a work W. FIG. A shop floor is a factory where a given product is manufactured. The work W is a product manufactured in this factory or a part attached to this product. The work W is transported in the transport direction D by the belt conveyor C at the work site.

In this embodiment, the user U of the display control system includes both an operator who wears the head mounted display 40 and an operator who does not. User U may be used.

The head mounted display 40 has a display 41 and a camera 42 . The display 41 is a transmissive or non-transmissive display device provided to cover the field of view of the user U when worn. The camera 42 is provided so as to capture at least part of the field of view of the user U when worn.

The imaging device 30 is a camera that captures images, and is installed at the work site so that the work W and the user U are included in the imaging range. A two-dimensional image in which a predetermined pattern is drawn in a square is attached to the workpiece W as a marker M. FIG. This marker M is a so-called rectangular marker used for detecting the position and orientation of the workpiece W to be added. Also, the marker M encodes a type ID that uniquely indicates the type of the workpiece W to be added. In this embodiment, the imaging device 30 shall be installed so that the marker M and the face of the user U can be imaged. Instead of the marker M, the type ID may be identified from the features of the work W or the production plan.

The two projection devices 20 are installed at the work site so as to be able to project an image onto the workpiece W conveyed by the belt conveyor C in a partial area including the conveying path. Also, the two projection devices 20 have different projection areas in the transport direction D, and are installed at the work site so that they partially overlap in the transport direction D. As shown in FIG. In the present embodiment, the two projectors 20 are assumed to be provided even in a state where their positions are fixed, but they may be attached movably.

The terminal device 50 is a terminal for performing various settings for the display control device 10, and is a laptop or desktop personal computer in this embodiment.

(Configuration of display control device)
The hardware configuration and functional configuration of the display control device will be described. 2 and 3 are block diagrams showing the hardware configuration and functional configuration of the display control device according to this embodiment, respectively.

As shown in FIG. 2, the display control device 10 includes, as hardware, a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage device 13, an input/output I/F (Interface) 14, a network I/ Equipped with F15.

The CPU 11 and RAM 12 cooperate to execute various functions, and the storage device 13 stores various data used for processing executed by the various functions. The input/output I/F 14 is an interface for inputting/outputting data with the two projection devices 20 and imaging device 30 . The network I/F 15 is an interface for performing wired communication and wireless communication with other devices such as the head mounted display 40 and the terminal device 50 .

As shown in FIG. 3, the display control device 10 includes an image recognition unit 101, a content conversion unit 102, an information management unit 103, a content selection unit 104, and a display control unit 105 as functions.

The image recognition unit 101 detects the marker M and the user U in the image included in the video imaged by the imaging device 30 . The content conversion unit 102 converts design information of the work W such as CAD (Computer Aided Design) data into content to be displayed by the projection device 20 or the head mounted display 40 .

The content can include any information that can be displayed by a known display device, and in this embodiment includes at least one of text, 2D (Dimension)/3D images, and 2D/3D video. CAD data is converted into content by extracting part of the CAD data as content based on preset extraction rules. Note that the extraction rule may be generated by machine learning based on the result of manual conversion from CAD data to content. In addition, the display control device 10 handles not only converted content but also content added/changed to the converted content and newly created content.

Contents are prepared for each type of work W and for each skill level of the user U, which will be described later. Furthermore, content including text is prepared for each of a plurality of languages assumed to be understandable by the user U. It is assumed that there is Specific examples of the content include a stereoscopic image/video showing the attachment positions of parts to the work W, manuals, images, and videos explaining work on the work W, notes on work, and the like.

The information management unit 103 manages user information indicating attributes of the user U, work information indicating attributes of the work W, and content information including at least one or more pieces of content. The content selection unit 104 selects content included in content information based on user information and work information. The display control unit 105 causes the projection device 20 or the head mounted display 40 to display the content selected by the content selection unit 104 . Note that the selection of content by the content selection unit 104 may be made based on the type of device used by the user U to browse content, based on the value of HMDID in the user information described later. Also, the content may be selected based on an initial value set prior to the user U using the display control system, or may be selected according to the user U's operation in the display control system.

(Various information)
User information, work information, and content information will be described.

The user information is information in which a user ID, skill level, placement position, first language, usable language, and HMDID are associated with each other. The user ID is an identifier that uniquely identifies the user U, and is, for example, an employee number assigned by the company where the user U works. The skill level indicates the skill level of the user U regarding work on the work W in stages. The placement position indicates the position where the user U is placed in the work site. The first language indicates the language that can be understood most easily among the languages that the user U can use. The usable language indicates languages other than the first language among the languages that the user U can use. HMDID is an identifier that uniquely indicates the head mounted display 40 worn by the user U, and is set to a null value when the user U does not wear the head mounted display 40 .

The work information is information in which a type ID and a work ID are associated with each other. The type ID is an identifier that uniquely indicates the type category to which the work W belongs, and is the product number or model number of the work W, for example. A work ID is an identifier that uniquely indicates a work W, such as a lot number or a serial number.

The content information is information in which a content ID, a type ID, a skill level, and a language used are associated with each other. A content ID is an identifier that uniquely indicates a content. The type ID indicates the type of work W associated with the content. The skill level indicates the skill level of the user U who is the target of content display. The used language indicates the language used in the content, and is set to a null value when the content does not contain a language.

User information, work information, and content information are all information preset as a master prior to operation of the display control system. is input to the display control device 10.

(Overall operation)
The overall operation of the display control device will be described. FIG. 4 is a flowchart showing the overall operation of the display control device according to this embodiment. It should be noted that this overall operation is performed at predetermined intervals.

As shown in FIG. 4, first, the image recognition unit 101 executes image recognition processing, which will be detailed later (S101). According to this image recognition processing, the work W and the user U are recognized in the image captured by the imaging device 30 .

Next, the content selection unit 104 selects an unselected user U among the users U recognized in the image recognition processing (S102), and executes content selection processing, which will be described in detail later (S103). According to this content selection process, the content displayed to the selected user U is selected.

Next, the display control unit 105 executes display control processing, which will be detailed later (S104). According to this display control process, the content regarding the work W is displayed by the projection device 20 or the head mounted display 40 .

Next, the content selection unit 104 determines whether or not all users U recognized in the image recognition process have been selected (S105). If all of the recognized users U are selected (S105, YES), the overall operation of the current period ends and the overall operation of the next period is started. On the other hand, if not all of the recognized users U are selected (S105, NO), the content selection unit 104 selects unselected users U (S102).

(Image recognition processing)
The operation of image recognition processing will be described. FIG. 5 is a flowchart showing the operation of image recognition processing.

As shown in FIG. 5, first, the image recognition unit 101 recognizes the workpiece W and the user U included in the captured image captured by the imaging device 30 as recognition targets (S201). Here, the image recognition unit 101 recognizes the work W by detecting features of the work W registered in advance and markers M, and also recognizes the user U by known human body detection that detects a human body from an image.

Next, the image recognition unit 101 selects an unselected recognition target from among the recognized recognition targets (S202), and determines whether or not the selected recognition target is the workpiece W (S203).

If the recognition target is the work W (S203, YES), the image recognition unit 101 identifies the type of the work W (S204), and determines whether or not all recognition targets have been selected (S205). In this embodiment, the image recognition unit 101 identifies the type of the workpiece W by detecting the type ID encoded in the marker M. FIG. The type of work W may be identified based on a feature amount relating to the appearance of the work W, or may be identified by any known method.

If all recognition targets have been selected (S205, YES), the image recognition unit 101 terminates the image recognition processing. On the other hand, if all recognition targets have not been selected (S205, NO), the image recognition unit 101 selects unselected recognition targets from among the recognized recognition targets (S202).

Further, in step S203, if the recognition target is not the work W (S203, NO), the image recognition unit 101 identifies the user U (S206), and determines whether or not all recognition targets have been selected ( S205). In this embodiment, the image recognition unit 101 performs personal recognition of the user U by face recognition or detection of an identification code attached to an item worn by the user U, such as an employee ID card. The personal identification of the user U may be performed by providing an RFID reader at the work site and causing the user U to read an RFID (Radio Frequency Identifier) tag on which the user ID is recorded with the IC reader. Personal identification of the user U may be performed by any known method, as with identification of the type of work W. FIG.

(Content selection processing)
The operation of content selection processing will be described. FIG. 6 is a flow chart showing the operation of content selection processing.

As shown in FIG. 6, first, the content selection unit 104 acquires user information including the user ID indicating the selected user U from among the user information managed by the information management unit 103 (S301).

In addition, the content selection unit 104 selects at least one work W from among the identified works W as a target work (S302), and indicates the type of the target work in the work information managed by the information management unit 103. Work information including the type ID is acquired (S303). In this embodiment, the content selection unit 104 selects the work W closest to the selected user U on the captured image or the work W positioned within a predetermined range from the arrangement position in the acquired user information. Select as

Next, the content selection unit 104 refers to the content information managed by the information management unit 103, and selects the type ID in the acquired work information, the skill level of the user U in the acquired user information, the first language, and the available languages. Based on this, content is selected (S304).

Specifically, the content selection unit 104 selects content information including the type ID in the acquired work information and the skill level in the acquired user information. Furthermore, when the language used in the selected content information is not null, that is, when the content indicated by the content ID included in the content information includes text, the content selection unit 104 selects the first language in the acquired user information. Select content information including content information or content information including available languages in the acquired user information.

According to such a content selection process, it is possible to display the content related to the work W corresponding to the skill level of the user U and the readable language.

(Display control processing)
The operation of display control processing will be described. FIG. 7 is a flowchart showing the operation of display control processing. FIG. 8 is a schematic diagram showing setting ranges.

As shown in FIG. 7, first, the display control unit 105 determines whether or not the selected user U is wearing the head mounted display 40 (S401). Here, the display control unit 105 refers to the HMDID included in the user information about the user U being selected, determines that the user U is wearing the head mounted display 40 when the HMDID exists, and determines that the HMDID is empty. value, it is determined that the user U is not wearing the head mounted display 40 .

When the selected user U is not wearing the head mounted display 40 (S401, NO), the display control unit 105 selects the unselected projection device 20 (S402), and the target work exists within the set area. (S403).

The set area is an area set for each projection device 20 , and is an area set so as to fit within a projection area that can be projected by the projection device 20 . The two projection devices 20 whose projection areas are adjacent in the transport direction D are provided so that the projection ranges P1 and P2 in the transport direction D at least partially overlap, as shown in FIG. Further, the set regions S1 and S2 of the two projection devices 20 are set in advance so as to be adjacent to each other without overlapping in the transport direction D. As shown in FIG.

In the present embodiment, the display control unit 105 determines that the target work exists within the set area when the entire marker M of the target work is within the set area of the projection device 20 being selected. Note that it may be determined that the target work exists within the set area when at least a portion of the marker M is within the set area or when at least a portion of the target work is within the set area. The criterion for determining whether or not the target work is within the set area is appropriately set according to various environments such as the work site, the set position of the projection device 20, and the work W recognition method.

If the target work exists within the set area (S403, YES), the display control unit 105 causes the currently selected content to be projected on the currently selected projection device 20 (S404). Here, when the content being selected is a 3D image/video, the display control unit 105 uses a known technique such as so-called projection mapping to convert it into a 2D image or video according to the three-dimensional shape of the target work. The projected content is projected on the projection device 20 .

On the other hand, if the target work does not exist in the settings (S403, NO), the display control unit 105 selects an unselected projection device 20 again (S402).

Further, in step S401, if the selected user U is wearing the head-mounted display 40 (S401, YES), the display control unit 105 selects the head mounted display indicated by the HMDID included in the user information about the selected user U. The mount display 40 is selected (S405).

Next, the display control unit 105 displays the content being selected on the head mounted display 40 being selected (S406). Here, if the display 41 is of a transmissive type, the display control unit 105 causes the head-mounted display 40 to display content corresponding to the target work in the real space. Further, when the display 41 is non-transmissive, the display control unit 105 causes the head mounted display 40 to display content corresponding to the target work on the image captured by the camera 42 . Further, when the selected content is a 3D image/video, the display control unit 105 causes the display 41 to display the content converted into a two-dimensional image or video according to the three-dimensional shape of the target work imaged by the camera 42. .

According to such display control processing, it is possible to display the content according to the display device that can be used by the selected user U. Content can be displayed at the corresponding position.

(display content)
Display content will be explained. FIG. 9 is a schematic diagram showing displayed content.

According to the display control system described above, a plurality of contents associated with the same work ID and with the same skill level relate to the same type of work W and are intended for users U with the same skill level. compose a set of content. Since this content set is simultaneously displayed to the user U in the above-described display control process, the user U views the content set as one content in which multiple contents included in this content set are superimposed or combined. can be visually recognized.

Also, the user U can determine whether or not to display each of the plurality of contents included in the content set. As a result, only the content required by the user U is displayed as the display content in the content set, and the visibility of the content can be improved.

While embodiments of the invention have been described, these embodiments are provided by way of example and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and its equivalents.

10 display control device 101 image recognition unit 103 information management unit 104 content conversion unit 105 display control unit

Claims (6)

A display control device for displaying content at a work site where a user performs predetermined processing on a work,
an image recognition unit that recognizes a workpiece based on a captured image of the work site;
an information management unit that manages content information in which a type of work and content related to the work belonging to the type are associated;
a content selection unit that selects content associated with the recognized work type based on the content information;
A display control device, comprising: a display control unit that causes a display device to display the selected content on the work site. The image recognition unit further recognizes the user based on the captured image,
The information management unit further manages user information that associates a user with a skill level indicating the skill of the user in stages,
The content information associates the type of work with the skill level,
The content selection unit selects content associated with the recognized work type and the recognized user's recognition level based on the user information and the content information. The display control device according to claim 1. The user information associates the user, the skill level of the user, and the language used, which is the source of the readability of the user,
The content information associates a type of work, a content related to the work belonging to the type, the skill level, and the language used,
The content selection unit associates the recognized work type, the recognized user's recognition level, and the recognized user's language used based on the user information and the content information. 3. The display control device according to claim 2, wherein the selected content is selected. The display device is a plurality of projection devices, and the work is transported at the work site,
For each of the plurality of projection devices, the display control unit determines whether or not a work exists within a set range set smaller than the projection range of the projection device, and determines whether or not the work exists within a set range set smaller than the projection range of the projection device. 4. The display control device according to any one of claims 1 to 3, wherein the selected content is displayed on the display control device. A display control program for displaying content at a work site where a user performs predetermined processing on a work,
the computer,
an image recognition unit that recognizes a workpiece based on a captured image of the work site;
an information management unit that manages content information in which a type of work and content related to the work belonging to the type are associated;
a content selection unit that selects content associated with the recognized work type based on the content information;
A display control program that functions as a display control unit that displays the selected content on a display device at the work site. A display control method for displaying content at a work site where a user performs predetermined processing on a work,
the computer
Recognize the workpiece based on the captured image of the work site,
managing content information in which a type of work and content related to the work belonging to the type are associated with each other;
selecting content associated with the recognized work type based on the content information;
A display control method for displaying the selected content on a display device at the work site.

Images