JP7492904B2 - Display device, display system, and display method - Google Patents

Description

The present invention relates to a display device, a display system, and a display method for displaying information to assist with work.

When manufacturing products in factories (plants) or inspecting or repairing factory equipment, workers may work while looking at work instructions such as procedure manuals or drawings. However, it can be difficult to place a device such as a display that displays the work instructions near the work object. In such cases, see-through head-mounted displays (hereinafter also referred to as HMDs) and smart glasses, which are worn on the worker's head and display images of a virtual space superimposed on real space, have attracted attention as display devices that can be used. By using a see-through HMD, workers do not need to hold a display device in their hands or go to look at a distant display device, improving work efficiency.

Display control in an HMD makes it easy to use by changing and configuring the display screen according to the state of the HMD or the worker, or the display content. For example, in the HMD described in Patent Document 1, an image taken by a virtual camera that captures a virtual space is generated according to the line of sight angle measured by the HMD, and is displayed together with an image that does not change according to the angle.

JP 2019-101330 A

In the invention described in Patent Document 1, for example, an image of a virtual space is displayed on a non-transparent HMD, and an image is generated that gives the worker (the person wearing the HMD) the sensation of being in the virtual space. On the other hand, in an HMD that supports work at a factory or other on-site, it is necessary to work while viewing the actual work object, tools, and surroundings of the site in the real space, so the technology described in Patent Document 1 cannot be said to be suitable as a display device for on-site work.
The present invention has been made in consideration of the above background, and has an objective of providing a display device, display system, and display method that are suitable for workers working in real space and that are capable of displaying information related to the work.

In order to solve the above-mentioned problems, the display device of the present invention comprises a display, a detector for detecting the orientation of the display, and an image display processing unit for displaying a first image in a predetermined first display area on the screen of the display, and displaying a second image related to the first image in a second display area on the screen of the display, the image display processing unit determines the presence or absence of the second display area on the screen of the display and, if present, its display position and display size according to the orientation of the display, the first display area is an area of a predetermined display position and display size independent of the orientation of the display , the first image is a predetermined image that is continuously displayed, and the second image is an image different from the first image, and is displayed continuously as a partial image according to the orientation of the display when the second display area is present .

The present invention provides a display device, display system, and display method that are suitable for workers working in real space and that enable display of work-related information. Problems, configurations, and effects other than those described above will become clear from the description of the embodiments below.

1 is a diagram showing a configuration of a display system according to a first embodiment. FIG. 2 is a functional block diagram of the HMD according to the first embodiment. FIG. 2 is a functional block diagram of the smartphone according to the first embodiment. FIG. 2 is a data structure diagram of a working document database according to the first embodiment. FIG. 2 is a diagram showing the configuration of a display source image according to the first embodiment; FIG. 2 is a diagram showing the configuration of a field of view image displayed on a display according to the first embodiment. FIG. 11 is a sequence diagram of a placement definition process according to the first embodiment. 11 is a flowchart of a display source image generating process according to the first embodiment. 11 is a flowchart of a field of view image display process in the first embodiment. FIG. 11 is a functional block diagram of an HMD according to a second embodiment. FIG. 11 is a functional block diagram of a smartphone according to a second embodiment. FIG. 11 is a data configuration diagram of a procedure manual coordinate table within a field of view according to the second embodiment. FIG. 13 is a sequence diagram (1) of the placement definition process according to the second embodiment. FIG. 13 is a sequence diagram (2) of the placement definition process according to the second embodiment. 13 is a flowchart of a display source image generating process according to the second embodiment. 13 is a flowchart of a visual field image display process according to the second embodiment. FIG. 13 is a data structure diagram of a working document database according to the third embodiment. 13 is a flowchart of a display source image generating process according to the third embodiment. 13 is a flowchart of a field of view image display process in the third embodiment. FIG. 13 is a diagram showing a configuration of a display system according to a fourth embodiment. FIG. 13 is a functional block diagram of an HMD according to a fourth embodiment. FIG. 13 is a functional block diagram of a smartphone according to a fourth embodiment. FIG. 13 is a diagram showing the configuration of a display source image according to the fourth embodiment. FIG. 13 is a diagram showing the configuration of a field of view image displayed on a display according to a fourth embodiment. FIG. 13 is a sequence diagram of a placement definition process according to the fourth embodiment. 13 is a flowchart of a display source image generating process according to the fourth embodiment. 13 is a flowchart of a field of view image display process in the fourth embodiment. FIG. 13 is a functional block diagram of an HMD according to a fifth embodiment. 13 is a diagram for explaining a split-display original image area according to the fifth embodiment; FIG. 13 shows a data configuration of a layout area table stored in an HMD according to the fifth embodiment. FIG. 13 is a functional block diagram of a smartphone according to a fifth embodiment. 13 is a diagram for explaining a layout area table stored in a smartphone according to a fifth embodiment. FIG. FIG. 13 is a data configuration diagram of a layout area table stored in a smartphone according to the fifth embodiment. FIG. 13 is a sequence diagram of a placement definition process according to the fifth embodiment. FIG. 13 is a screen configuration diagram of a procedure manual placement area setting screen according to the fifth embodiment. 13A to 13C are diagrams for explaining changes in the positions of the procedure manual placement area and the drawing placement area according to the fifth embodiment. 13A to 13C are diagrams for explaining changes in size of the procedure manual placement area and the drawing placement area according to the fifth embodiment. 13 is a flowchart of a display source image generating process according to the fifth embodiment. 13A and 13B are diagrams for explaining a display original image of a change mode displayed on a touch panel display according to a fifth embodiment. 13 is a diagram for explaining a display original image in the unchangeable mode displayed on a touch panel display according to a fifth embodiment. FIG. 13 is a flowchart of a field of view image display process in the fifth embodiment. 13 is a diagram showing how a screen displayed on a display looks to a worker wearing an HMD according to a fifth embodiment. FIG. 13 is a diagram showing how a screen displayed on a display looks when a worker wearing an HMD according to a fifth embodiment looks to the right. FIG.

Below, a display system in a form (embodiment) for implementing the present invention is described. The display system is configured to include a smartphone and a see-through HMD. The smartphone stores work instructions and drawings, and generates an image (also referred to as a display source image) that includes the instructions and drawings of the work step being performed by the worker in their respective specified areas, and transmits it to the HMD. The HMD cuts out an image from the area of the instructions included in the display source image, and displays it at a specified position (e.g., bottom center) on the HMD screen.

The HMD also displays the drawing depending on the orientation of the device. For example, assume that the drawing area is on the right side of the displayed original image (see drawing arrangement area 232 in displayed original image 230 in FIG. 5 described below). Then, the HMD will not display the drawing when the worker is facing forward. When the HMD detects that the worker has turned slightly to the right, it will display the left side of the drawing on the right side of the screen, and when it detects that the worker has turned significantly to the right, it will display the entire drawing on the right side of the screen.

In this way, by displaying the procedure manual and drawings on the HMD, the worker can work while constantly referring to the procedure manual. Also, by turning to the right, the worker can refer to the drawings. When facing forward, the drawings are not displayed, and the worker's view of the work object or work site is not obstructed. As a result, the worker can work while referring to the procedure manual and drawings while working with both hands, without having to operate a smartphone or HMD with their hands.

First embodiment: Overall configuration of display system
FIG. 1 is a diagram showing a configuration of a display system 10 according to a first embodiment. The display system 10 includes an HMD (head mounted display) 100 (display device) and a smartphone 200 (portable device). The HMD 100 is, for example, a goggle-type display device worn on the head so as to cover the field of vision. The HMD 100 and the smartphone 200 are connected, for example, by a USB (Universal Serial Bus) cable, but may be connected by a cable of another standard or wirelessly. In addition, a USB may be connected to the smartphone 200, and converted to an image output connector such as an HDMI (High-Definition Multimedia Interface) or a DisplayPort via a conversion adapter, and connected to the HMD 100. The HMD 100 also includes smart glasses that are worn like glasses. Similarly, the smartphone 200 (portable device) also includes portable terminals such as a tablet or a notebook PC.

First embodiment: Configuration of HMD (head mounted display)
2 is a functional block diagram of the HMD 100 according to the first embodiment. The HMD 100 includes a control unit 110 including a central processing unit (CPU), a storage unit 120 including a read only memory (ROM), a random access memory (RAM), a flash memory, etc., a display 181, a sensor 182, and a communication unit (not shown). The communication unit includes one or more communication interfaces such as USB and Wi-Fi (registered trademark), and transmits and receives data to and from the smartphone 200.

The display 181 is a transmissive display device with high transmittance provided in front of the HMD 100 (see FIG. 1). In addition to the binocular type as shown in FIG. 1, a monocular type used with either the left or right eye may also be used. The worker can wear the HMD 100 and work while referring to information displayed on the display 181.
The sensor 182 is, for example, a sensor including a MEMS (Micro Electro Mechanical Systems) type gyroscope, which detects the angle or angular velocity of the HMD 100 and outputs it to the control unit 110. The sensor 182 may include an illuminance sensor that measures the illuminance of the surroundings, and a camera that captures images of the surroundings and the worker.

The storage unit 120 stores a program 121, a display source image 122, a procedure manual coordinates 123, related drawing coordinates 124, and procedure manual coordinates within the field of view 125. The program 121 is a program executed by a CPU constituting the control unit 110, and controls the HMD 100. The program 121 includes a field of view image display process (see FIG. 9 described later) and other processing procedures.
The display original image 122 is a display original image 230 (see Figs. 3 and 5 described later) received from the smartphone 200, and is image data that is the source of the visual field image displayed on the display 181. The instruction manual coordinates 123, the associated drawing coordinates 124, and the in-visual-field instruction manual coordinates 125 are coordinate data indicating an area in the display original image 122 received from the smartphone 200, and are equivalent to the instruction manual coordinates 222, the associated drawing coordinates 223, and the in-visual-field instruction manual coordinates 224 stored in the smartphone 200 (see Fig. 3 described later), respectively.

The control unit 110 includes a data receiving unit 111 and a field of view image display unit 112. The data receiving unit 111 stores the data received from the smartphone 200 in the display original image 122, the instruction manual coordinates 123, the related drawing coordinates 124, and the instruction manual coordinates within the field of view 125. The field of view image display unit 112 references the instruction manual coordinates 123, the related drawing coordinates 124, and the instruction manual coordinates within the field of view 125 to cut out an image from the display original image 122 (the display original image 230 shown in FIG. 5 described below) and displays it on the display 181.

First embodiment: Configuration of smartphone
3 is a functional block diagram of the smartphone 200 according to the first embodiment. The smartphone 200 includes a control unit 210 including a CPU, a storage unit 220 including a ROM, a RAM, a flash memory, etc., a touch panel display 281, a microphone 282, and a communication unit (not shown). The communication unit includes one or more communication interfaces such as USB and Wi-Fi, and transmits and receives data to and from the HMD 100.

The memory unit 220 stores a program 221, an original display image 230, procedure manual coordinates 222, related drawing coordinates 223, procedure manual coordinates within the field of view 224, a work manual database 240 (referred to as a work manual DB (database) in Figure 3, see Figure 4 described below), and a drawing database 250 (referred to as a drawing DB in Figure 3).
The program 221 is executed by the CPU constituting the control unit 210, and controls the smartphone 200. The program 221 includes a display source image generating process (see FIG. 8 described later) and other processing procedures.

Figure 4 is a data structure diagram of the workbook database 240 according to the first embodiment. The workbook database 240 is, for example, data in a table format. One row of the workbook database 240 indicates one work step performed by a worker who is a user of the display system 10, and includes a step number 241 (indicated as # in Figure 4), a work target area 242, work content 243, related drawings 244, a completion flag 245 (indicated as completion F in Figure 4), and a completion date and time 246.

Step number 241 is a number assigned to a work step, and indicates the order of the work step. Work target area 242 indicates the area that is the subject of work such as repair or inspection, for example, the name of the work target, such as "Model A No. 1". Work content 243 is an explanation of the work step displayed on display 181 (see Figures 1 and 2), for example, an explanation of the work content, such as "Turn off switch B". Related drawing 244 is a drawing related to the work step, and is identification information of the drawing displayed on display 181. Completion flag 245 indicates whether the work step has been completed ("YES") or not ("NO"). Completion date and time 246 is the date and time when the work step was completed.

Returning to FIG. 3, the drawing database 250 stores drawings related to work steps. Identification information corresponding to the related drawings 244 (see FIG. 4) is assigned to the drawings, and the control unit 210 can access the drawings by specifying the identification information. The source image 230, the procedure manual coordinates 222, the related drawing coordinates 223, and the procedure manual coordinates within the field of view 224 will be described later with reference to FIG. 5 and FIG. 6.

First embodiment: original image and field of view image
5 is a diagram showing the configuration of the display source image 230 according to the first embodiment. The display source image 230 is image data that is the source of an image displayed on the display 181 (see FIGS. 1 and 2). The procedure manual placement area 231 is an area in the display source image 230 where the contents of the procedure manual (step number 241, work target area 242, work content 243, related drawing 244, and completion flag 245 in FIG. 4) are displayed. The drawing placement area 232 is an area in the display source image 230 where drawings related to the procedure manual (drawings identified by related drawing 244 in FIG. 4 and stored in drawing database 250) are displayed.

The visual field area 233 is a partial area of the original image 230, and is an area displayed on the display 181 (see Figures 1 and 2), and the size of the area is the display size of the display 181. The visual field area 233 moves within the original image 230 according to the orientation of the HMD 100. For example, when the worker turns to the left and the sensor 182 (see Figure 2) detects a left rotation of the HMD 100, the visual field area 233A on the left side of the original image 230 is displayed on the display 181.

The procedure manual coordinates 222 (see FIG. 3) are the placement coordinates of the procedure manual placement area 231 in the source image 230. The placement coordinates are, for example, the coordinates of the upper left vertex and lower right vertex of the area in the source image 230. The placement coordinates may be the coordinates of the upper left vertex of the area, and the size of the area. The related drawing coordinates 223 are the placement coordinates of the drawing placement area 232 in the source image 230.

Figure 6 is a diagram showing the configuration of a field of view image 420 displayed on the display 181 according to the first embodiment. The field of view image 420 is an image corresponding to the field of view area 233 (see Figure 5), and is an image in which a field of view procedure area 421 has been added to an image in which a field of view drawing area 422 (described below) has been cut out from the original display image 230. On the display 181, pixels are not displayed in any part other than the field of view drawing area 422 and the field of view procedure area 421, so that a worker wearing the HMD 100 can see the scenery of the work site through that part of the field of view.

The in-field-of-view procedure manual area 421 is a partial area of the field of view image 420, and is the area where the procedure manual is displayed. In the first embodiment, the in-field-of-view procedure manual area 421 is positioned at the lower center of the field of view image 420, but it may be positioned in another position. The in-field-of-view procedure manual coordinates 224 (see FIG. 3) are the position coordinates of the in-field-of-view procedure manual area 421 in the field of view image 420.

The in-field drawing area 422 is an area where the field of view 233 (see FIG. 5) and the drawing placement area 232 overlap, and is an area that includes part of the drawing. The field of view 233 moves within the original image 230 depending on the orientation of the HMD 100, so the in-field drawing area 422, which is an area where the field of view 233 and the drawing placement area 232 overlap, also changes depending on the orientation of the HMD 100. For example, when the worker turns to the left, he moves from the field of view 233 to the field of view 233A. Then, the field of view 233A and the drawing placement area 232 do not overlap, so the in-field drawing area 422 does not exist and no drawing is displayed on the display 181.

First embodiment: Configuration of smartphone: control unit
Returning to FIG. 3, the control unit 210 includes a placement definition unit 211 and a display source image generation unit 212. The placement definition unit 211 receives an instruction from a worker who is a user of the display system 10, and defines (sets) the placement (placement coordinates) of the procedure manual placement area 231 and the drawing placement area 232 in the display source image 230 (see FIG. 5), and the placement of the procedure manual area 421 in the field of view in the field of view image 420 (see FIG. 6). The placement definition unit 211 transmits the defined placement coordinates to the HMD 100. The display source image generation unit 212 generates a display source image 230 including a procedure manual and related drawings related to the work step in which the worker is working, and transmits the generated image to the HMD 100.

First embodiment: placement definition process
Fig. 7 is a sequence diagram of the placement definition process according to the first embodiment. Processing of the smartphone 200 and the HMD 100 that configure settings related to the display source image 230 (see Fig. 5) and the visual field image 420 (see Fig. 6) will be described with reference to Fig. 7.
In step S111, the placement definition unit 211 of the smartphone 200 acquires placement coordinates of the procedure manual placement area 231 in the original display image 230 (see FIG. 5 ) instructed by the operator who is the user of the display system 10, and stores the coordinates in the procedure manual coordinates 222 (see FIG. 3 ). The placement coordinates are, for example, the coordinates in the original display image 230 of the upper left vertex and the lower right vertex of the area.

In step S<b>112 , the placement definition unit 211 acquires the placement coordinates of the drawing placement area 232 in the original display image 230 designated by the worker, and stores them in the related drawing coordinates 223 .
In step S113, the placement definition unit 211 acquires the placement coordinates of the in-field-of-view procedure manual area 421 in the field-of-view image 420 (see FIG. 6) designated by the worker, and stores them in the in-field-of-view procedure manual coordinates 224.
In step S<b>114 , the placement definition unit 211 transmits the procedure manual coordinates 222 , the related drawing coordinates 223 , and the procedure manual coordinates within the field of view 224 to the HMD 100 .

In step S115, the data receiving unit 111 of the HMD 100 stores the received instruction manual coordinates 222, associated drawing coordinates 223, and instruction manual coordinates within the field of view 224 in the instruction manual coordinates 123, associated drawing coordinates 124, and instruction manual coordinates within the field of view 125, respectively.
Following the placement definition process, the process of the display system 10 when a worker uses the display system 10 to perform work will be described.

First embodiment: source image generation process
Fig. 8 is a flowchart of a display source image generating process in the first embodiment. A process in which the smartphone 200 generates a display source image 230 and transmits it to the HMD 100 will be described with reference to Fig. 8 .
In step S131, the display source image generating unit 212 of the smartphone 200 proceeds to step S132 if there is an instruction to complete the work step (step S131→YES), and proceeds to step S133 if there is no instruction to complete the work step (step S131→NO). The instruction to complete the work step is an instruction to the display system 10 to record that the worker has completed one work step. The display source image generating unit 212 may determine whether or not there is an instruction to complete the work step by detecting the worker's utterance of "Work step completed" from a sound acquired by the microphone 282 (see FIG. 3). Alternatively, the display source image generating unit 212 may detect the worker's tapping of a work step completion button displayed on the touch panel display 281 and determine that there is an instruction to complete the work step.

In step S132, the display source image generating unit 212 records the completion of the work step. More specifically, the display source image generating unit 212 updates the completion flag 245 (see FIG. 4) of the current work step to "YES" and the completion date and time 246 to the current time. The current work step is the work step whose completion flag 245 is "NO" and whose step number 241 is the smallest.
In step S 133 , the display source image generating unit 212 generates a blank image and stores it in the display source image 230 .

In step S134, the display source image generating unit 212 draws the procedure manual in the procedure manual placement area 231 (see FIG. 5 ) of the display source image 230. In detail, the display source image generating unit 212 draws in the procedure manual placement area 231 the step number 241 of the current work step, the work target area 242, the work content 243, the related drawing 244, and the completion flag 245 (see FIG. 4 ).
In step S135, the display source image generating unit 212 draws the related drawing in the drawing arrangement area 232 of the display source image 230. In detail, the display source image generating unit 212 retrieves a drawing corresponding to the identification information in the related drawing 244 of the current work step from the drawing database 250 and draws it in the drawing arrangement area 232.
In step S136, the display original image generating unit 212 transmits the display original image 230 to the HMD 100, and the process returns to step S131.

First embodiment: visual field image display processing
Fig. 9 is a flowchart of the visual field image display process in the first embodiment. The process of displaying the visual field image on the display 181 based on the display source image 230 (see step S136 in Fig. 8 ) received by the HMD 100 from the smartphone 200 will be described with reference to Fig. 9 .
In step S<b>151 , the data receiving unit 111 of the HMD 100 receives the display original image 230 transmitted by the smartphone 200 and stores it in the display original image 122 .
In step S152, the field of view image display unit 112 of the HMD 100 displays an image of the instruction manual placement area 231 (see Figure 5, a partial area of the displayed original image 122 indicated by the instruction manual coordinates 123) in the area of the display 181 indicated by the in-field of view instruction manual coordinates 125 (see in-field of view instruction manual area 421 in Figure 6).

In step S<b>153 , the visual field image display unit 112 calculates the visual field area 233 (see FIG. 5 ) from the orientation (angle) of the HMD 100 acquired by the sensor 182 .
In step S154, the field of view image display unit 112 displays the overlapping area between the field of view area 233 (see FIG. 5) and the drawing arrangement area 232 in the corresponding area of the display 181, and returns to step S151. The corresponding area of the display 181 is an area that corresponds to the in-field drawing area 422 in the field of view image 420 when the display image of the display 181 is regarded as the field of view image 420 (see FIG. 6).

First embodiment: Features of display system
The display 181 of the HMD 100 displays the procedure manual for the work step in which the worker is working and related drawings. The procedure manual is always displayed at a set position on the display 181 (see the procedure manual area 421 in the field of view in FIG. 6). This allows the worker to work while always referring to the procedure manual.
The related drawing is displayed according to the direction of the worker's head (HMD 100). It is assumed that the position of the drawing is set to the right side of the original image 230. Then, when the worker faces forward, the drawing is not displayed, and the worker can work without being disturbed by the drawing. The worker can refer to the drawing by turning to the right. As a result, the worker can refer to the procedure manual and the drawing while working with both hands without operating the smartphone 200 or the HMD 100 with his/her hands.

Second Embodiment
In the first embodiment, the procedure manual is always displayed in the procedure manual area 421 in the field of view image 420 (see FIG. 6), but the display/non-display may be switched by the operator's instruction. Also, the display area of the procedure manual is fixed at the center lower side (see the procedure manual area 421 in the field of view in FIG. 6), but the display/non-display and the display position may be switched by the operator's instruction. Regarding the drawing, the drawing in the display source image 230 may not be specified using the drawing arrangement area 232 specified by coordinates, but a background color may be specified so that pixels other than the background color are treated as the drawing. Below, a second embodiment will be described that enables the display/non-display and display position of the procedure manual and the designation of the background color. Note that, like the procedure manual, the display/non-display of the drawing may be switched by the operator's instruction.

Second embodiment: configuration of HMD
FIG. 10 is a functional block diagram of the HMD 100A according to the second embodiment. Compared with the HMD 100 according to the first embodiment (see FIG. 2), the related drawing coordinates 124 are eliminated in the storage unit 120, and a background color 126 and a manual display flag 127 are added. The background color 126 is the color that becomes the background of the drawing, and the pixels of the color other than the background color 126 in the display source image 122 become the drawing. The manual display flag 127 is a flag indicating whether or not to display the manual on the display 181. The data receiving unit 111A receives the background color and the manual display flag in addition to the display source image and the manual coordinates from the smartphone 200 and stores them in the storage unit 120. The field of view image display unit 112A refers to the background color 126 and the manual display flag 127 to display the manual and the drawing on the display 181.

Second embodiment: Smartphone configuration
11 is a functional block diagram of a smartphone 200A according to the second embodiment. Compared to the smartphone 200 of the first embodiment, a visual field instruction manual coordinate table 260 (see FIG. 12 described later) and a visual field instruction manual coordinate identification number 269 are stored in the storage unit 220 instead of the visual field instruction manual coordinate 224. In addition, a background color 225 corresponding to the background color 126 and the instruction manual display flag 127, and an instruction manual display flag 226 are added.

Figure 12 is a data configuration diagram of the in-field procedure manual coordinate table 260 according to the second embodiment. The in-field procedure manual coordinate table 260 stores the placement coordinates of one or more in-field procedure manual areas 421 (see Figure 6). The in-field procedure manual coordinate table 260 is, for example, data in a table format, and one row (record) indicates one in-field procedure manual coordinate, and includes a column (attribute) of the in-field procedure manual coordinate identification number 261 (indicated as # in Figure 12) and the in-field procedure manual coordinate 262. The in-field procedure manual coordinate identification number 261 is identification information of the in-field procedure manual coordinate. The in-field procedure manual coordinate 262 indicates the placement coordinate of the in-field procedure manual area 421. The in-field procedure manual coordinate identification number 269 (see Figure 11) is the in-field procedure manual coordinate identification number 261 of the in-field procedure manual area 421 (see Figure 6) specified by the operator.

Returning to FIG. 11, the placement definition unit 211A accepts instructions from the worker and defines the placement and background color 225 of the procedure manual placement area 231. The display source image generation unit 212A generates a display source image 230 including the procedure manual and related drawings, and transmits it to the HMD 100A.

Second embodiment: placement definition process
Fig. 13 is a sequence diagram (1) of the placement definition process according to the second embodiment. Fig. 14 is a sequence diagram (2) of the placement definition process according to the second embodiment. In the second embodiment, the placement definition process is performed in such a manner that the process of Fig. 13 is followed by the process of Fig. 14 which is repeatedly executed during work.

In step S211, the placement definition unit 211A of the smartphone 200A acquires placement coordinates of the procedure manual placement area 231 in the original display image 230 (see FIG. 5) specified by the operator who is the user of the display system 10, and stores them in the procedure manual coordinates 222.
In step S212, the placement definition unit 211A acquires the placement coordinates of the drawing placement area 232 in the original image 230 designated by the worker, and stores them in the related drawing coordinates 223.

In step S213, the placement definition unit 211A acquires the placement coordinates of the in-field procedure manual area 421 in the field of view image 420 (see FIG. 6) specified by the worker, and stores them in the in-field procedure manual coordinate table 260 (see FIG. 12). The number of in-field procedure manual areas 421 is not limited to one, and may be multiple. In the second embodiment, the worker is assumed to have specified three procedure manual placement areas (see FIG. 12). The placement definition unit 211A stores "1" as the default in-field procedure manual coordinate identification number in the in-field procedure manual coordinate identification number 269, and stores "ON" as the default procedure manual display flag in the procedure manual display flag 226. The placement definition unit 211A may inquire of the worker about the default in-field procedure manual coordinate identification number and the procedure manual display flag, and store them in the in-field procedure manual coordinate identification number 269 and the procedure manual display flag 226.

In step S214, the placement definition unit 211A obtains the background color of the drawing designated by the worker, and stores it in the background color 225.
In step S215, the placement definition unit 211A transmits the instruction manual coordinates 222, the instruction manual coordinates in the field of view 262 (see FIG. 12) corresponding to the instruction manual coordinate identification number 269 in the field of view, the background color 225, and the instruction manual display flag 226 to the HMD 100A.
In step S216, the data receiving unit 111A of HMD 100A stores the received instruction manual coordinates 222, instruction manual coordinates in the field of view 262 corresponding to the instruction manual coordinate identification number 269 in the field of view, background color 225, and instruction manual display flag 226 in the instruction manual coordinates 123, instruction manual coordinates in the field of view 125, background color 126, and instruction manual display flag 127, respectively.

14, in step S221, if there is an instruction to display the procedure manual (step S221→YES), the placement definition unit 211A proceeds to step S222, and if there is no instruction (step S221→NO), the placement definition unit 211A returns to step S221. The instruction to display the procedure manual is whether or not to display the procedure manual, and the display position. For example, the worker issues an instruction to display the procedure manual by saying "display procedure manual" or "procedure manual display ON". The worker also issues an instruction to display the procedure manual by saying "procedure manual display position 2" or "display procedure manual at position 2". The placement definition unit 211A may determine whether or not there is an instruction from a sound acquired by the microphone 282. Alternatively, the placement definition unit 211A may determine whether or not there is an instruction by detecting a tap on the procedure manual display ON/OFF button or procedure manual display position button displayed on the touch panel display 281 by the worker.

In step S222, if the instruction in step S221 is an instruction to display the procedure manual (step S222→YES), the placement definition unit 211A proceeds to step S223, and if not (step S222→NO), the placement definition unit 211A proceeds to step S224.
In step S223, the placement definition unit 211A sets the procedure manual display flag 226 (see FIG. 11) to "ON."

In step S224, if the instruction in step S221 is an instruction to not display the procedure manual (step S224->YES), the placement definition unit 211A proceeds to step S225, and if not (step S224->NO), the placement definition unit 211A proceeds to step S226.
In step S225, the placement definition unit 211A sets the procedure manual display flag 226 to "OFF."

In step S226, if the instruction in step S221 is to make the display position of the procedure manual the first one (step S226→YES), the placement definition unit 211A proceeds to step S227, and if the instruction is not to make it the first one (step S226→NO), the placement definition unit 211A proceeds to step S228.
In step S227, the placement definition unit 211A sets the procedure manual coordinate identification number within field of view 269 to "1."

In step S228, if the instruction in step S221 was to change the display position of the procedure manual to number 2 (step S228→YES), the placement definition unit 211A proceeds to step S229, and if the instruction was not to change the display position to number 2 (step S228→NO), the placement definition unit 211A proceeds to step S230.
In step S229, the placement definition unit 211A sets the procedure manual coordinate identification number within field of view 269 to "2".

In step S230, if the instruction in step S221 is to change the display position of the procedure manual to number 3 (step S230→YES), the placement definition unit 211A proceeds to step S231, and if the instruction is not to change the display position to number 3 (step S230→NO), the placement definition unit 211A proceeds to step S232.
In step S231, the placement definition unit 211A sets the procedure manual coordinate identification number within field of view 269 to "3."

In step S232, the placement definition unit 211A transmits the in-field-of-view instruction manual coordinates 262 corresponding to the in-field-of-view instruction manual coordinate identification number 269 and the instruction manual display flag 226 to the HMD 100A, and returns to step S221.
In step S233, the data receiving unit 111A of the HMD 100A stores the in-field-of-view instruction manual coordinates 262 and instruction manual display flag 226 corresponding to the received in-field-of-view instruction manual coordinate identification number 269 in the in-field-of-view instruction manual coordinates 125 and instruction manual display flag 127, respectively.
While the worker is working, the placement definition unit 211A and the data receiving unit 111A repeat the process of FIG.

Second embodiment: source image generation process
Fig. 15 is a flowchart of a display source image generating process in the second embodiment. A process in which the smartphone 200A generates a display source image 230 and transmits it to the HMD 100A will be described with reference to Fig. 15 .
Steps S241 and S242 are similar to steps S131 and S132 (see FIG. 8).

In step S 243 , the display source image generating unit 212 A generates a blank image that is the same color as the background color 225 , and stores it in the display source image 230 .
In step S244, if the instruction manual display flag 226 is "ON" (step S244->YES), the display source image generating unit 212A proceeds to step S245, and if not (step S244->NO), the display source image generating unit 212A proceeds to step S246.
Steps S245 to S247 are similar to steps S134 to S136.

Second embodiment: visual field image display processing
Fig. 16 is a flowchart of a visual field image display process in the second embodiment. A process in which the HMD 100A generates a visual field image from the display source image 230 and displays it on the display 181 will be described with reference to Fig. 16 .
Step S261 is the same process as step S151 (see FIG. 9).
In step S262, if the procedure manual display flag 127 is "ON" (step S262->YES), the field of view image display unit 112A proceeds to step S263, and if it is not "ON" (step S262->NO), the field of view image display unit 112A proceeds to step S265.

Step S263 is the same process as step S152.
In step S264, the field of view image display unit 112A extracts pixels other than the background color 126 from the original display image 122 other than the procedure manual placement area 231 (see FIG. 5) as related drawings.
In step S265, the field of view image display unit 112A extracts pixels other than the background color 126 from the entire display source image 122 as related drawings.
Step S266 is the same process as step S153 (see FIG. 9).
In step S267, the field of view image display unit 112A of the HMD 100A displays the overlapping area between the field of view 233 and the pixels extracted as the related drawing in step S264 or step S265 in the corresponding area of the display 181, and returns to step S261. In other words, the field of view image display unit 112A displays the pixels that are in the field of view 233 and that are extracted as the related drawing in step S264 or step S265 in the corresponding area of the display 181, and returns to step S261.

Second embodiment: Features of display system
Compared to the first embodiment, the HMD 100A can switch the display/non-display of the instruction manual displayed on the display 181 and switch the display position according to the operator's instructions. Therefore, the operator can select non-display when he/she wants to secure a wider field of view, or can switch the display position of the instruction manual according to the work. The operator can easily secure a field of view according to the work step and the operator's posture, etc., improving work efficiency. In addition, since the display portion of the drawing is specified by color instead of coordinates, it becomes possible to display figures of complex shapes, not limited to simple shapes such as rectangles.

Third embodiment
In the second embodiment, the worker can change the position of the procedure manual displayed on the display 181. Below, a third embodiment will be described in which the position of the drawing can be changed depending on the work step.

FIG. 17 is a data configuration diagram of the worksheet database 240B according to the third embodiment. Compared with the worksheet database 240 (see FIG. 4) according to the first embodiment, a column (attribute) of related drawing coordinates 247 is added. The related drawing coordinates 247 indicate the arrangement coordinates of the drawing arrangement area 232 in the original image 230 (see FIG. 5). In the second embodiment, all drawing arrangement areas 232 are the same regardless of the work step (drawing), but in the third embodiment, the drawing arrangement area 232 can be set for each work step. Also, in the second embodiment, the background color 225 is the same regardless of the work step (drawing), but in the third embodiment, a background color is selected for each drawing.
Although not shown, in the third embodiment, reference numerals are given to components such as a display system 10B, an HMD 100B, a display source image generating unit 212B, and a visual field image display unit 112B.

Third embodiment: source image generation process
18 is a flowchart of the display source image generation process in the third embodiment. Steps S341 to S347, excluding steps S343, S346, and S347, are similar to steps S241 to S247 (see FIG. 15), excluding steps S243, S246, and S247.

In step S343, the display source image generating unit 212B selects the color least used in the related drawings of the current work step as the background color, generates a blank image of this color, and stores it in the display source image 230. The display source image generating unit 212B also stores this color in the background color 225.
In step S346, the display source image generating unit 212B refers to the work document database 240B and draws the related drawing at the related drawing coordinates 247 (see FIG. 17) of the current work step in the display source image 230.
In step S347, the display original image generating unit 212B transmits the display original image 230 and the background color 225 to the HMD 100B.

Third embodiment: visual field image display processing
19 is a flowchart of the visual field image display process in the third embodiment. Steps S361 to S367 except for step S361 are similar to steps S261 to S267 (see FIG. 16) except for step S261.
In step S361, the field of view image display unit 112B receives the display original image 230 and the background color 225, and stores them in the display original image 122 and the background color 126 (see FIG. 10), respectively.

Third embodiment: Features of display system
Compared to the second embodiment, the worker can set the display position of each work step (drawing) in advance. This allows the worker to determine the direction of the HMD 100B (worker's face) on which the drawing is displayed depending on the work part, improving the usability of the display system 10B.
Furthermore, the smartphone 200B selects the least used color in each drawing as the background color, which eliminates the need for the worker to set the background color, as compared to the second embodiment.

Fourth embodiment
In the above-described embodiment, a manual or a drawing is displayed on the display 181. Information other than the manual or the drawing may be displayed on the display 181. In the following, a fourth embodiment will be described in which the battery state of the smartphone, the time, and a message to be described later are displayed.

Figure 20 is a diagram showing the configuration of a display system 10C according to a fourth embodiment. The display system 10C includes an HMD 100C, a smartphone 200C, and an administrator terminal 500. The HMD 100C has a camera 183 on the front for capturing images of the work being performed. The smartphone 200C and the administrator terminal 500 are connected using wireless communication, for example, a fifth-generation mobile communication system. The administrator using the administrator terminal 500 monitors the work of the workers and, as necessary, sends messages in the form of text information to instruct the workers on the work or to call their attention.

Figure 21 is a functional block diagram of the HMD 100C according to the fourth embodiment. The sensor 182C includes a camera 183 in addition to a gyroscope. The instruction manual/battery state coordinates 123C, clock coordinates 128, and in-field instruction manual/battery state coordinates 125C will be described later with reference to Figures 23 and 24. The data receiving unit 111C receives the instruction manual/battery state coordinates and clock coordinates in addition to the original image to be displayed, and stores them in the memory unit 120. The field of view image display unit 112C displays the time, battery state, and messages in addition to the instruction manual and drawings, and transmits images from the camera 183 to the smartphone 200C.

Figure 22 is a functional block diagram of the smartphone 200C according to the fourth embodiment. The display source image 230C, clock coordinates 227, battery status coordinates 228, instruction manual/battery status coordinates 229, in-field instruction manual/battery status coordinates 224C, and in-field clock coordinates 270 will be described later with reference to Figures 23 and 24. The message buffer 271 holds messages sent from the administrator terminal 500. The placement definition unit 211C sets the placement of the battery status and clock in addition to the instruction manual and drawings, and sends the placement coordinates to the HMD 100C. The display source image generation unit 212C generates a display source image including a clock (time) and a message and sends it to the HMD 100C, and also receives an image of the camera 183 from the HMD 100C and sends it to the administrator terminal 500.

Figure 23 is a diagram showing the configuration of a display original image 230C according to the fourth embodiment. The differences from the display original image 230 (see Figure 5) are explained below. The clock layout area 234 (shown as C in Figure 23) is an area in the display original image 230C where the clock (time) is displayed. The battery status layout area 235 (shown as B in Figure 23) is an area in the display original image 230C where the battery status (remaining battery power) of the smartphone 200C is displayed. The instruction manual/battery status layout area 236 is an area combining the instruction manual layout area 231 and the battery status layout area 235.

The clock coordinates 227 (see FIG. 22) are the coordinates of the clock placement area 234 in the original image 230C. The battery status coordinates 228 are the coordinates of the battery status placement area 235 in the original image 230C. The instruction manual/battery status coordinates 229 are the coordinates of the instruction manual/battery status placement area 236 in the original image 230C.

24 is a diagram showing the configuration of the field of view image 420C displayed on the display 181 according to the fourth embodiment. The in-field procedure manual/battery status area 421C is a partial area of the field of view image 420C, and is an area where the procedure manual and the battery status are displayed side by side. If there is a message from the administrator terminal 500, the message is displayed in the in-field procedure manual/battery status area 421C instead of the procedure manual and the battery status. The in-field clock area 424 is a partial area of the field of view image 420C, and is an area where a clock is displayed.
The in-view instruction manual/battery status coordinates 224C (see FIG. 22) are the location coordinates of the in-view instruction manual/battery status area 421C in the field of view image 420C. The in-view clock coordinates 270 are the location coordinates of the in-view clock area 424 in the field of view image 420C.

Fourth embodiment: placement definition process
FIG. 25 is a sequence diagram of the placement definition process according to the fourth embodiment.
In step S411, the placement definition unit 211C acquires placement coordinates of the procedure manual placement area 231 in the display source image 230C (see FIG. 23) designated by the worker who is the user of the display system 10C, and stores them in the procedure manual coordinates 222.
In step S412, the placement definition unit 211C acquires the placement coordinates of the drawing placement area 232 in the display source image 230C designated by the worker, and stores them in the related drawing coordinates 223.

In step S413, the arrangement definition unit 211C acquires the arrangement coordinates of the battery state arrangement area 235 in the display source image 230C designated by the operator, and stores them in the battery state coordinates 228.
In step S414, the placement definition unit 211C stores the placement coordinates of the procedure manual/battery status placement area 236 in the procedure manual/battery status coordinates 229. Note that the procedure manual/battery status placement area 236 is an area that combines the procedure manual placement area 231 and the battery status placement area 235.

In step S415, the placement definition unit 211C acquires the placement coordinates of the in-field procedure manual/battery status area 421C in the field of view image 420C (see Figure 24) specified by the worker, and stores them in the in-field procedure manual/battery status coordinates 224C.
In step S416, the placement definition unit 211C acquires the placement coordinates of the clock placement area 234 in the display source image 230C designated by the worker, and stores them in the clock coordinates 227.
In step S417, the placement definition unit 211C obtains the background color of the drawing designated by the worker, and stores it in the background color 225.

In step S418, the placement definition unit 211C transmits the instruction manual/battery state coordinates 229, the associated drawing coordinates 223, the clock coordinates 227, the instruction manual/battery state coordinates within the field of view 224C, and the background color 225 to the HMD 100C.
In step S419, the data receiving unit 111C stores the received instruction manual/battery status coordinates 229, associated drawing coordinates 223, clock coordinates 227, instruction manual/battery status coordinates within the field of view 224C, and background color 225 in instruction manual/battery status coordinates 123C, associated drawing coordinates 124, clock coordinates 128, instruction manual/battery status coordinates within the field of view 125C, and background color 126, respectively.

<Fourth embodiment: display source image generation processing>
26 is a flowchart of a display source image generation process in the fourth embodiment. A major difference from the first and second embodiments is the process performed when a message arrives from the administrator terminal 500.
Steps S431 to S433 are similar to steps S241 to S243 (see FIG. 15).

In step S434, if the display source image generating unit 212C has received a message from the administrator terminal 500 (step S434→YES), the process proceeds to step S435, and if not (step S434→NO), the process proceeds to step S437.
In step S435, the display source image generating unit 212C stores the received message in the message buffer 271.
In step S436, the display source image generating unit 212C draws the message in the message buffer 271 in the procedure manual/battery state placement area 236 (see FIG. 23) of the display source image 230C, and proceeds to step S441.

In step S437, if the display source image generating unit 212C receives an instruction to delete the message from the administrator terminal 500 (step S437->YES), the process proceeds to step S438, and if not (step S437->NO), the process proceeds to step S439.
In step S438, the display source image generating unit 212C deletes the message stored in the message buffer 271.
In step S439, if there is a message in the message buffer 271 (step S439→YES), the display source image generating unit 212C proceeds to step S436, and if there is no message in the message buffer 271 (step S439→NO), the display source image generating unit 212C proceeds to step S440.

In step S440, the display source image generation unit 212C draws the instruction manual, which is the work content 243 (see Figure 4) of the current work step, and the battery status in the instruction manual placement area 231 and the battery status placement area 235 (see Figure 23) of the display source image 230C, respectively, and proceeds to step S441.
Step S441 is similar to step S135 (see FIG. 8).
In step S442, the display source image generating unit 212C draws a clock (time) in the clock arrangement area 234 of the display source image 230C.

In step S443, the display original image generating unit 212C transmits the display original image 230C to the HMD 100C.
In step S444, the display source image generating unit 212C receives the image acquired by the camera 183 from the HMD 100C, transmits it to the administrator terminal 500, and returns to step S431. Note that the process in which the HMD 100C transmits the image of the camera 183 corresponds to step S456 in FIG. 27 described later.

<Fourth embodiment: visual field image display processing>
27 is a flowchart of the visual field image display process in the fourth embodiment. A major difference from the visual field image display process in the second embodiment (see FIG. 16) is the process of transmitting the image of the camera 183 (see step S456).
Step S451 is the same process as step S151 (see FIG. 9).
In step S452, the field of view image display unit 112C displays the instruction manual/battery status location area 236 (see Figure 23, a partial area of the displayed original image 122 indicated by the instruction manual/battery status coordinates 123C) in the area of the display 181 specified by the in-field of view instruction manual/battery status coordinates 125C.

In step S453, the field of view image display unit 112C extracts pixels other than the background color 126 from the display source image 122 other than the procedure manual/battery state location area 236 as related drawings.
Step S454 is the same process as step S153.
In step S455, the field of view image display unit 112C displays an overlapping area between the field of view 233 and the pixels extracted as the related drawing in step S453 in the corresponding area of the display 181. In other words, the field of view image display unit 112C displays the pixels extracted as the related drawing in step S453 within the field of view 233 in the corresponding area of the display 181.
In step S456, the field of view image display unit 112C transmits the image captured by the camera 183 to the smartphone 200C, and the process returns to step S451.

<<Fourth embodiment: Features of display system>>
In the fourth embodiment, in addition to the procedure manual and the drawings, the battery state and the time of the smartphone 200C are displayed on the display 181. This allows the worker to check the battery state and the time of the smartphone 200C without interrupting the work.
Although the manager using the manager terminal 500 is not at the site, he or she can know the status of the work from the images captured by the camera 183. In addition, the manager can send messages to give instructions to the workers or support the work by informing them of important points, enabling the workers to proceed with the work efficiently.

Fifth embodiment
In the above-described embodiment, the operator can change the position of the contents such as the instruction manual and the drawing displayed on the display 181 of the HMD 100 to 100C. In the first embodiment, the smartphone 200 transmits to the HMD 100 the arrangement coordinates of the instruction manual arrangement area 231, the drawing arrangement area 232, and the in-field-of-view instruction manual area 421 in advance in the arrangement definition unit 211 (see FIG. 3) (see step S114 in FIG. 7).

In the second embodiment, the placement definition unit 211A transmits to the HMD 100A not only the placement coordinates but also the instruction manual display flag 226 (see FIG. 11) indicating whether the instruction manual is displayed or not (see step S215 in FIG. 13). In the fourth embodiment, the placement definition unit 211C transmits to the HMD 100C the placement coordinates of the clock placement area 234 (see FIG. 23) and the instruction manual/battery status placement area 236 (see step S418 in FIG. 25).

In the fifth embodiment, the display source image 230 (see Figs. 5 and 23) is divided into a plurality of areas (hereinafter also referred to as split display source image areas), and the worker selects the procedure manual placement area 231 from among these split display source image areas. This allows the worker to easily specify the area in which to place the procedure manual. In addition, by simplifying the communication procedure between the HMD 100D (see Fig. 28 described below) and the smartphone 200D (see Fig. 31 described below), implementation becomes easier.

In addition, in the display source image 230 generated by the smartphone 200D, the background color other than the instruction manual and the drawings is set to a color that the HMD 100D transmits (for example, R value = G value = B value = 0). This makes it unnecessary for the HMD 100D to consider the placement coordinates of the instruction manual, drawings, etc., making implementation easier.

The fifth embodiment is based on the following (1) to (4).
(1) The aspect ratio of the display 181 of the HMD 100D is 16:9.
(2) The aspect ratio of the touch panel display 281 of the smartphone 200D is 18:9.
(3) The size (dimension, number of pixels) of the display source image 230 (see FIG. 5) is 3×3 times (three times both vertically and horizontally) that of the display 181. The aspect ratio of the display source image 230 is the same as that of the display 181, that is, 16:9.
(4) The horizontal length (number of pixels) of the original image 230 and the touch panel display 281 are the same.
Note that the above premise is an example and is not limited to this. For example, the aspect ratio may be different from 16:9 or 18:9. Furthermore, the size ratio of the display source image 230 to the display 181 or the touch panel display 281 may be different from the above.

Fifth embodiment: configuration of HMD
28 is a functional block diagram of the HMD 100D according to the fifth embodiment. Compared to the HMD 100 according to the first embodiment (see FIG. 2), instead of the instruction manual coordinates 123, an instruction manual area number 123D is stored in which the identification number of the divided display original image area in which the instruction manual is arranged is stored. In addition, the related drawing coordinates 124 and the in-field instruction manual coordinates 125 are eliminated, and a placement area table 130 (see FIG. 30 described later) and a background color 126 are added. The background color 126 is the same as that of the second embodiment (see FIG. 10), and is the color that becomes the background of the instruction manual or the drawing, and the pixels of the color other than the background color 126 in the display original image 230 become the instruction manual or the drawing.

The data receiving unit 111D receives from the smartphone 200D the display source image 230 and the identification number of the split display source image area in which the instruction manual is placed, and stores them in the display source image 122D and the instruction manual area number 123D, respectively. The field of view image display unit 112D references the placement area table 130 and the instruction manual area number 123D, and displays the instruction manual and the drawing on the display 181.

Figure 29 is a diagram for explaining split display original image areas according to the fifth embodiment. The split display original image areas are areas obtained by dividing the display original image 230 into 3 x 3 areas. The numbers 1 to 9 are assigned to the split display original image areas as identification numbers. The size (size, number of pixels vertically and horizontally) of each split display original image area is the same as the size of the display 181 (see Figure 1) of the HMD 100D (see premise (3) of the fifth embodiment described above), with a pixels horizontally and b pixels vertically.

30 shows the data configuration of the placement area table 130 stored in the HMD 100D according to the fifth embodiment. The placement area table 130 is data in a table format, in which one row (record) indicates one divided display source image area and includes columns (attributes) of number 131, coordinates 132, and size 133.
Number 131 is an identification number of the split display original image area. Coordinates 132 are the coordinates of the split display original image area, and are the coordinates of the upper left vertex of the split display original image area in the display original image 230. Note that the upper left of the display original image 230 is the origin (0,0). Size 133 is the size of the split display original image area.

Fifth embodiment: Configuration of smartphone
Fig. 31 is a functional block diagram of a smartphone 200D according to the fifth embodiment. Compared to the smartphone 200A (see Fig. 11) according to the second embodiment, the storage unit 220 does not have the in-field-of-view procedure manual coordinate table 260 and the in-field-of-view procedure manual coordinate identification number 269, but instead has a placement area table 290 (see Fig. 32 described later) and a drawing display flag 226A.

The instruction manual display flag 226 and the drawing display flag 226A are flags that respectively set whether or not the instruction manual and the drawing are displayed on the display 181 of the HMD 100D. In the second embodiment, the instruction manual display flag 226 is transmitted to the HMD 100A (see step S215 in FIG. 13 and step S232 in FIG. 14), but in the fifth embodiment, the instruction manual display flag 226 and the drawing display flag 226A are not transmitted (see step S515 in FIG. 34 described below).

FIG. 32 is a diagram for explaining the layout area table 290 (see FIG. 33 described later) stored in the smartphone 200D according to the fifth embodiment. Each area numbered 1 to 9 is a split display original image area, and has a size of a pixels horizontally and b pixels vertically. The area with Dx pixels horizontally and Dy pixels vertically, excluding the two hatched areas (padding) from the nine split display original image areas, corresponds to the touch panel display 281 (see FIG. 31) of the smartphone 200D, and Dx:Dy=18:9. The height of the upper and lower paddings is P/2, respectively. The value of P is set to be Dx:Dy+P=16:9=a:b, which is the aspect ratio of the display original image 230.
The above-mentioned positions and sizes of padding are merely examples. For example, padding may be placed only on the left and right, or only on the top, bottom, left and right. Padding may not be required. For example, when the aspect ratios of the original image 230, the display 181, and the touch panel display 281 are the same, padding may not be required.

Fig. 33 is a data configuration diagram of a layout area table 290 stored in a smartphone 200D according to the fifth embodiment. The layout area table 290 is data in a tabular format, with one row (record) indicating a split-display source image area on one touch panel display 281, and including columns (attributes) of number 291, coordinates 292, and size 293.

Number 291 is an identification number of the split display original image area. Coordinates 292 are the coordinates of the split display original image area on touch panel display 281, and are the coordinates of the upper left vertex of the split display original image area on touch panel display 281 in original image 230. The upper left vertex of original image 230 is the origin (0,0). Size 293 is the size of the split display original image area on touch panel display 281.
The height of the split display original image areas with identification numbers 1 to 3 and 7 to 9 is (b-P/2) pixels. Moreover, the coordinates 292 of the split display original image area on touch panel display 281 with identification number 1 is (0, P/2).

Returning to FIG. 31, the placement definition unit 211D accepts instructions from the worker and sets an area in the placement area table 290 for placing the procedure manual as the procedure manual placement area 231 (see FIG. 5). The display source image generation unit 212D generates a display source image 230 including the procedure manual and related drawings, and transmits it to the HMD 100D.

Fifth embodiment: placement definition process
FIG. 34 is a sequence diagram of the placement definition process according to the fifth embodiment.
In step S511, the placement definition unit 211D of the smartphone 200D displays a procedure manual placement area setting screen 600 (see FIG. 35 described later) on the touch panel display 281 to prompt the worker to select a split display source image area in which to place the procedure manual.

Figure 35 is a screen configuration diagram of a procedure manual placement area setting screen 600 according to the fifth embodiment. The procedure manual placement area setting screen 600 includes a map 601 showing the display source image 230 divided into nine split display source image areas. The worker selects the area in which to place the procedure manual by tapping on any of the split display source image areas. In Figure 35, the split display source image area with identification number 4 has been tapped and is highlighted to indicate that it has been selected. In the following explanation, it is assumed that the split display source image area with identification number 4 has been selected.

34, in step S512, the placement definition unit 211D acquires the background color designated by the worker and stores it in the background color 225D (see FIG. 31). In the following description, it is assumed that R value=G value=B value=0 is designated.
In step S513, the placement definition unit 211D obtains the appropriateness of the display of the procedure manual and the appropriateness of the display of the related drawings instructed by the worker, and stores them in the procedure manual display flag 226 and the drawing display flag 226A.

In step S514, the placement definition unit 211D displays the display source image 230 (see FIG. 36 described later) on the touch panel display 281, and accepts changes to the position and size of the procedure manual placement area 231 and the drawing placement area 232 (see FIG. 36 and FIG. 37 described later). Upon accepting the changes, the placement definition unit 211D stores the placement coordinates of the procedure manual placement area 231 in the procedure manual coordinates 222D, and stores the placement coordinates of the drawing placement area 232 in the associated drawing coordinates 223D.

Figure 36 is a diagram for explaining the change in position of the procedure manual placement area 231 and drawing placement area 232 according to the fifth embodiment. The dashed lines on the display source image 230 indicate the boundaries of the split display source image areas. In step S511, the split display source image area 239 with identification number 4 was selected as the area in which to place the procedure manual. Therefore, at the start of step S514, the procedure manual placement area 231 is placed in the split display source image area 239 (lightly hatched area). The drawing placement area 232 is placed in the other split display source image area (darkly hatched area).

The placement definition unit 211D detects a swipe operation on the procedure manual placement area 231, and accepts an instruction to move the procedure manual placement area 231. Note that the procedure manual placement area 231 can be moved within the divided display source image area 239 with identification number 4.
The arrangement definition unit 211D detects a swipe operation on the drawing arrangement area 232, and receives an instruction to move the drawing arrangement area 232. Note that the drawing arrangement area 232 can be moved within other divided display source image areas (dark hatched areas).

Figure 37 is a diagram for explaining the resizing of the procedure manual placement area 231 and the drawing placement area 232 according to the fifth embodiment. The placement definition unit 211D detects a swipe operation on a vertex or edge of the procedure manual placement area 231, and accepts an instruction to resize the procedure manual placement area 231. Note that the procedure manual placement area 231 can be enlarged within the split display source image area 239 with identification number 4. Figure 37 shows an operation in which the bottom right vertex of the procedure manual placement area 231 is swiped to enlarge it to area 231A.

The placement definition unit 211D detects a swipe operation on a vertex or edge of the drawing placement area 232, and accepts an instruction to change the size of the drawing placement area 232. Note that the drawing placement area 232 can be enlarged within other divided display source image areas (dark hatched areas). Fig. 37 shows an operation in which the bottom edge of the drawing placement area 232 is swiped to enlarge it to the area 232A.
In addition, the placement definition unit 211D accepts pinch-in and pinch-out operations as size change operations for the procedure manual placement area 231 and the drawing placement area 232.

Returning to FIG. 34, in step S515, the placement definition unit 211D transmits to the HMD 100D the identification number (4 in the current explanation) of the divided display source image area in which the instruction manual is placed and the background color.
In step S516, the data receiving unit 111D of the HMD 100D stores the received identification number and background color in the instruction manual area number 123D and background color 126, respectively.

Fifth embodiment: source image generation process
Fig. 38 is a flowchart of a display source image generating process in the fifth embodiment. A process in which the smartphone 200D generates a display source image 230 and transmits it to the HMD 100D will be described with reference to Fig. 38 .
Steps S521 and S522 are similar to steps S131 and S132.

In step S523, if the display state of the touch panel display 281 is in the change mode (see FIG. 39 described below) (step S523→YES), the display source image generation unit 212D proceeds to step S524, and if the display state is in the non-changeable mode (see FIG. 40 described below) (step S523→NO), the display source image generation unit 212D proceeds to step S525.

Fig. 39 is a diagram for explaining a display source image 230A in change mode displayed on a touch panel display 281 according to the fifth embodiment. Fig. 40 is a diagram for explaining a display source image 230B in non-changeable mode displayed on a touch panel display 281 according to the fifth embodiment. In comparison with the display source image 230B, the boundary of the split display source image area is drawn as a dashed line in the display source image 230A. The worker can distinguish between change mode and non-changeable mode depending on whether this boundary is displayed or not. The placement definition unit 211D accepts changes to the position and size of the procedure manual placement area 231 and the drawing placement area 232 in the change mode.

If no operation is performed on the touch panel display 281 within a predetermined time in the change mode, the mode transitions to the non-changeable mode. If an operation on the touch panel display 281 is detected in the non-changeable mode, the mode transitions to the change mode. The non-changeable mode may be transitioned to the change mode by another operation on the smartphone 200D, such as a voice or button operation.

Returning to FIG. 38, in step S524, the placement definition unit 211D performs the placement definition process (see FIG. 34).
In step S525, the display source image generating unit 212D generates a blank image that is the same color as the background color 225D, and stores it in the display source image 230.
In step S526, if the procedure manual display flag 226 is ON (step S526→ON), the display source image generating unit 212D proceeds to step S527, and if it is OFF, the display source image generating unit 212D proceeds to step S528.
In step S527, the display source image generation unit 212D draws the instruction manual in the instruction manual placement area 231 by referring to the placement coordinates of the instruction manual placement area 231 stored in the instruction manual coordinates 222D.

In step S528, if the drawing display flag 226A is ON (step S528→ON), the display source image generating unit 212D proceeds to step S529, and if it is OFF, the display source image generating unit 212D proceeds to step S530.
In step S529, the display source image generating unit 212D draws the related drawing in the drawing placement area 232 by referring to the placement coordinates of the drawing placement area 232 stored in the related drawing coordinates 223D.
In step S530, the display original image generating unit 212D transmits the display original image 230 to the HMD 100D, and the process returns to step S521.

Fifth embodiment: visual field image display processing
Fig. 41 is a flowchart of a visual field image display process in the fifth embodiment. A process in which the HMD 100D generates a visual field image from the display source image 122D and displays it on the display 181 will be described with reference to Fig. 41.
In step S541, the data receiving unit 111D of the HMD 100D receives the display original image 230 transmitted by the smartphone 200D and stores it in the display original image 122D (see FIG. 28).
In step S542, the field of view image display unit 112D of the HMD 100D calculates the field of view area 233 (see FIG. 5) from the orientation (angle) of the HMD 100 acquired by the sensor 182.

In step S543, the field of view image display unit 112D displays on the display 181 the pixels other than the background color 126 in the field of view area 233 from the original display image 230.
In step S544, the field of view image display unit 112D refers to the instruction manual area number 123D and the arrangement area table 130 (see FIG. 30) to identify the split display original image area in which the instruction manual is arranged in the display original image 230 stored in the display original image 122D. In the present explanation, the instruction manual area number 123D is 4, so the coordinates of the split display original image area are (0, b) and the size is (a, b).

In step S545, the field of view image display unit 112D displays pixels other than the background color 126 in the split display original image area identified in step S544 on the display 181. The positions at which the pixels are displayed in the display 181 are the same as the positions of the pixels in the split display original image area. In other words, the field of view image display unit 112D regards the split display original image area identified in step S544 as the field of view area, and displays pixels other than the background color 126 on the display 181. Note that the field of view image display unit 112D may use an alpha blending technique to superimpose and display the split display original image area and the field of view calculated in step S542.

Fifth embodiment: Features of display system
Compared with the first, second, and fourth embodiments, the fifth embodiment allows the user to specify the split-display source image area in which the instruction manual is to be placed, improving the usability of the display system. In addition, the HMD 100D can specify the area of the instruction manual by receiving only the identification number and background color of the split-display source image area in which the instruction manual is to be placed from the smartphone 200D (see step S515 in FIG. 34). This reduces the amount of data exchanged with the smartphone 200D, simplifies the configuration of the HMD 100D, and ultimately reduces development costs.
If the background color is fixed, there is no need to set the background color (see step S512 in FIG. 34). This eliminates the need to transmit the background color in step S515, further reducing the amount of data and development costs.

42 is a diagram showing how a screen displayed on the display 181 looks to a worker wearing the HMD 100D according to the fifth embodiment. The screen displayed on the display 181 is the visual field area 233 (see FIG. 5 ) in the original display image 230, and the aspect ratio of the screen is the same as that of the original display image 230.
If the instruction manual display flag 226 is ON, the instruction manual is always displayed on the display 181 (see step S545 in FIG. 41). On the other hand, the part of the related drawing that is displayed varies depending on the orientation of the operator (HMD 100D). In FIG. 42, a part of the lower left corner of the related drawing is displayed on the display 181 (the viewing area 233 includes a part of the lower left corner of the drawing arrangement area 232).

Figure 43 is a diagram showing how the screen displayed on the display 181 looks when a worker wearing an HMD 100D according to the fifth embodiment turns to the right. When the worker turns to the right, the viewing area 233 moves to the right in the original image 230. As a result, the lower half of the drawing in the drawing placement area 232 is included in the viewing area 233, and the worker can see the lower half of the drawing. Note that the procedure is always displayed, and is displayed in front of the drawing (above the drawing).

As explained above, frequently referenced procedure manuals are always displayed, while the visible range of less frequently referenced related drawings can be controlled by the movement of the worker's head. This improves the usability of the display system and improves the worker's work efficiency. Furthermore, if a procedure manual and a related drawing overlap, the worker can specify ON/OFF of the procedure manual display flag or the drawing display flag (see step S513 in Figure 34) to give priority to the display of one or the other (see steps S526 to S529 in Figure 38). This improves the visibility of procedure manuals, drawings, and the work site.

<<Variation: Specifying the work step>>
In the embodiment described above, it is assumed that the work steps are advanced in the order of step number 241 (see FIG. 4), and the work proceeds to the next work step upon receiving an instruction to complete the work (see step S131 in FIG. 8). The worker may specify a step number, and the smartphone 200 may generate a display source image including a procedure manual or a drawing for the work of the specified step number and transmit the image to the HMD.

<<Modification: HMD battery status>>
In the fourth embodiment, the battery status of the smartphone 200C is displayed on the display 181. The visual field image display unit 112C may acquire and display the battery status of the HMD 100C instead of the battery status of the smartphone 200C.

<<Variation: Background color>>
In the second embodiment, the related drawing is an image that is composed of pixels of a color other than the background color in the original image. Conversely, the color of the pixels that compose the related drawing may be specified, and an image composed of pixels of that color may be the related drawing. Multiple colors may be set.

<<Variation: Completion Record>>
The display system of the above embodiment records the completion date and time 246 (see FIG. 4) as a completion record of the work step. Images and videos captured by the HMD camera 183 (see FIG. 20) may be recorded together with the completion date and time. Alternatively, at the instruction of the worker, the smartphone may instruct the HMD to acquire images captured by the camera 183 and store them in association with the current work step. In this way, in addition to the date and time, a photo taken after the work is completed can be easily recorded, leaving evidence of the work.

<<Modification: Split display original image area and procedure manual placement area>>
In the fifth embodiment described above, the procedure manual placement area is included in one split display original image area. The procedure manual placement area may be included in a plurality of split display original image areas. For example, the original image may be divided into 10×10 split display original image areas, and one of the 2×2 split display original image areas may be used as the procedure manual placement area.

Other variations
Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. For example, the above-mentioned display system is composed of an HMD and a smartphone, but the HMD and the smartphone may be integrated. In addition, the instruction to complete the work is given by voice, but the instruction may be given by a button on the HMD, and the HMD may notify the smartphone that the button has been operated. The smartphone stores instructions and drawings, but the instructions and drawings may be obtained from a device different from the smartphone via a communication line.

The present invention may take various other embodiments, and various modifications such as omissions and substitutions may be made without departing from the spirit of the present invention. These embodiments and their modifications are included within the scope and spirit of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and their equivalents.

10, 10B, 10C Display system 100, 100A, 100B, 100C, 100D HMD (display device)
111, 111A, 111B, 111C, 111D Data receiving unit 112, 112A, 112B, 112C, 112D Field of view image display unit (image display processing unit)
181 Display (indicator)
182 Sensor (detector)
183 Camera (imaging device)
200, 200A, 200B, 200C, 200D Smartphone (portable device)
211, 211A, 211A, 211C, 211D Arrangement definition section 212, 212A, 212B, 212C, 212D Display source image generation section 230, 230C Display source image 231 Procedure manual arrangement area 232 Drawing arrangement area (second original image)
233, 233A Field of view area 239 Split display original image area 240 Operation manual database 250 Drawing database 420, 420C Field of view image 421 Procedure manual area in field of view (first display area)
421C Field of view procedure manual/battery status area (first display area)
422 Drawing area within the field of view (second display area)
500 Administrator terminal (remote terminal)

Claims (14)

A display device;
A detector for detecting an orientation of the display;
an image display processing unit that displays a first image in a predetermined first display area on a screen of the display device and displays a second image related to the first image in a second display area on the screen of the display device;
the image display processing unit determines, in accordance with an orientation of the display device , whether or not the second display area exists on the screen of the display device, and, if so, a display position and a display size of the second display area;
the first display area is an area having a predetermined display position and display size that are independent of the orientation of the display device,
the first image is a predetermined image that is continuously displayed ;
When the second display area exists, the second image is a partial image of an image different from the first image, the partial image being displayed continuously depending on the orientation of the display device.
A display device comprising: A data receiving unit is further provided for receiving a display source image, which is an image, from a portable device;
the first image is a first original image which is a partial image of the display original image,
The display device according to claim 1 , wherein the second image is a part of a second original image which is a partial image of the displayed original image. The display device according to claim 2 , wherein the first original image and the second original image are images in a region at a predetermined position of the displayed original image. The display device according to claim 2, characterized in that the first original image and the second original image are images consisting of pixels of a predetermined color contained in the display original image, or images consisting of pixels of a color different from the predetermined color contained in the display original image. The second image is
The display device according to claim 2, characterized in that the viewing area is an area that moves within the displayed original image depending on the orientation of the display device and is an area displayed on the screen of the display device, and is an overlapping portion of the second original image. the display device is a head-mounted display device,
the first image is a procedure manual for a task to be performed by a wearer of the display device,
The display device according to claim 1 , wherein the second image is a drawing related to the instruction manual. A display system including a display device and a mobile device,
The portable device includes:
a display original image generating unit that generates a display original image including a first original image and a second original image related to the first original image, and transmits the display original image to the display device;
The display device includes:
A display device;
A detector for detecting an orientation of the display;
an image display processing unit that displays the first original image in a predetermined first display area on a screen of the display device, and displays a partial image of the second original image or the second original image in a second display area on the screen of the display device;
the image display processing unit determines, in accordance with an orientation of the display device , whether or not the second display area exists on the screen of the display device, and, if so, a display position and a display size of the second display area;
the first display area is an area having a predetermined display position and display size that are independent of the orientation of the display device,
The first original image is a predetermined image that is continuously displayed ;
When the second display area exists, the image displayed in the second display area is a partial image of an image different from the first original image, the partial image being displayed continuously in accordance with the orientation of the display device.
A display system comprising: The display system according to claim 7 , wherein the portable device replaces at least one of the first original image and the second original image with another image when an instruction from a user of the display system is detected. The display device includes an imaging device.
The display system according to claim 8 , wherein, when the portable device detects the instruction, the portable device acquires the image captured by the imaging device and stores the image in association with the first original image and the second original image. Further comprising a remote terminal;
The display system according to claim 7 , wherein when the mobile device receives a message from the remote terminal, the display original image generating unit sets an image of the message as the first original image. The display device includes an imaging device.
the image display processing unit transmits an image captured by the imaging device to the portable device;
11. The display system of claim 10, wherein the portable device transmits received images to the remote terminal. A display method for a display device including a display and a detector for detecting an orientation of the display, comprising:
displaying a first image in a predetermined first display area on a screen of the display device, and displaying a second image related to the first image in a second display area on the screen of the display device;
determining whether the second display area exists on the screen of the display device and, if so, the display position and the display size of the second display area according to the orientation of the display device;
the first display area is an area having a predetermined display position and display size that are independent of the orientation of the display device,
the first image is a predetermined image that is continuously displayed ;
When the second display area exists, the second image is a partial image of an image different from the first image, the partial image being displayed continuously depending on the orientation of the display device.
A display method comprising: The source image is divided into a plurality of regions;
The display device according to claim 2 , wherein the first original image is included in at least one of the regions. The source image is divided into a plurality of regions;
The display system of claim 7 , wherein the first original image is included in at least one region of interest.

Images