WO2023218740A1 - Display control system and wearable device - Google Patents

Abstract

According to the present invention, this display control system comprises a wearable device and a terminal device. The wearable device includes a first display control unit that controls a first transmissive display device positioned in front of a first user, and a first communication control unit that transmits a captured image in which a real space included in the visual field range of the first user is captured to the terminal device. The terminal device includes a second communication control unit that receives the captured image, a second display control unit that displays the captured image on a second display device, and a reception unit that receives, from a second user, the designation of a first area in the captured image displayed on the second display device. The second communication control unit transmits area information specifying the range of the first area on the captured image to the wearable device. The first display control unit displays a first virtual object indicating a range corresponding to the first area in a real space on the first display device on the basis of the area information.

Description

Display control systems and wearable devices

The present invention relates to a display control system and a wearable device.

Conventionally, XR glasses to which XR (Extended Reality) technology, represented by AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) are applied, have been widespread. The XR glasses may be provided with an imaging device that captures an image in front of the user. A captured image captured by the imaging device is used, for example, to assist a user wearing XR glasses in his or her work. Alternatively, the captured image is used for recording, etc. for later evaluation of whether the user has performed the work appropriately.

For example, Patent Document 1 below discloses a work support system that uses markers placed in real space to align a virtual object placed in virtual space with an object in real space. Specifically, the work support system includes a display section, an imaging section, a position information detection section, and a display control section. The display unit displays an image of the virtual space corresponding to the real space. The imaging unit images a range including the center of the user's visual field. The positional information detection unit detects the positional relationship between the marker and the imaging unit based on the captured image of the marker. The display control unit controls the display of the image in the virtual space based on the detection result of the marker image detected while the user is looking at the marker and the detection result of mechanically detecting the user's posture. .

Japanese Patent Application Publication No. 2020-170249

In general, there is a gap between the imaging range of the imaging device mounted on the XR glasses and the user's field of view. Therefore, the imaging range assumed by the user and the actual imaging range do not match, and there is a possibility that a required captured image cannot be obtained. Furthermore, when a user performs a task while receiving advice from a remote assistant, for example, the user may not be able to grasp the location that the assistant wants to focus on, that is, the location that the assistant wants to capture with the imaging device. Work efficiency may decrease.

An object of the present invention is to efficiently capture an image depicting a specific range of real space.

A display control system according to one aspect of the present invention includes a wearable device and a terminal device, wherein the wearable device controls a transmissive first display device located in front of a first user. and a first communication control unit that transmits a captured image of a real space included in the field of view of the first user to the terminal device, and the terminal device a second communication control unit that receives an image; a second display control unit that displays the captured image on a second display device; and a second communication control unit that displays the captured image on the second display device; a reception unit that receives information from a user, the second communication control unit transmits area information that specifies the range of the first area in the captured image to the wearable device, and the first display control unit A first virtual object indicating a range corresponding to the first area in the real space is displayed on the first display device based on the area information.

A wearable device according to one aspect of the present invention includes a first display control unit that controls a transmissive first display device located in front of a first user's eyes, and a real space included in a visual field range of the first user that is imaged. a transmitter that transmits the captured image to a terminal device, and a receiver that receives area information that specifies a range of a first area specified in the captured image, and the first display control unit includes A first virtual object indicating a range corresponding to the first area in the real space is displayed on the first display device based on the area information.

According to one aspect of the present invention, an image depicting a specific range in real space can be efficiently captured.

FIG. 1 is a block diagram showing the configuration of a display control system 1 according to an embodiment. 1 is a diagram schematically showing a usage state of the wearable device 10. FIG. FIG. 2 is a diagram schematically showing how the terminal device 20 is used. 1 is a block diagram showing the configuration of a wearable device 10. FIG. 2 is a block diagram showing the configuration of a terminal device 20. FIG. FIG. 2 is an explanatory diagram schematically showing a range related to the visual field range R1 of the first user U1. 2 is a diagram showing an example of a captured image displayed on a display device 201. FIG. 2 is a diagram showing an example of a captured image displayed on a display device 201. FIG. 3 is a diagram showing an example of a display range R3 of the transmissive display device 101. FIG. 3 is a diagram showing an example of a display range R3 of the transmissive display device 101. FIG. 7 is a diagram showing another example of the display range R3 of the transmissive display device 101. FIG. 7 is a diagram showing an example of a display when a part of the first virtual frame V1-1 is removed from the display range R3. FIG. It is a figure which shows an example of a display when the ratio of range RY to display range R3 becomes below a predetermined value.

A. Embodiment A-1. System Configuration FIG. 1 is a block diagram showing the configuration of a display control system 1 according to an embodiment. The display control system 1 includes a wearable device 10 and a terminal device 20. Wearable device 10 and terminal device 20 are connected via communication network N.

FIG. 2 is a diagram schematically showing how the wearable device 10 is used. As shown in FIG. 2, the wearable device 10 is a head mounted display mounted on the head of a first user U1 who is a worker. The wearable device 10 includes a transmissive display device 101 (see FIG. 4) located in front of the eyes of the first user U1 wearing the wearable device 10. The transmissive display device 101 is an example of a first display device. The shape of the wearable device 10 is, for example, similar to general eyeglasses. Wearable device 10 includes a lens L and a frame F that supports lens L. The lens L includes a left lens placed in front of the left eye of the first user U1 and a right lens placed in front of the user's right eye. Frame F has a bridge provided between the left lens and the right lens, and a pair of temples that extend over the left and right ears. As described later, the transmissive display device 101 displays various images including virtual objects on the lens L. The lens L is the display range of the transmissive display device 101.

In this embodiment, the first user U1 performs wiring work between the plurality of devices DV1 to DV3 stored in the rack RA while wearing the wearable device 10. Wearable device 10 includes an imaging device 105 (see FIG. 4). Wearable device 10 transmits a captured image captured by imaging device 105 to terminal device 20 .

FIG. 3 is a diagram schematically showing how the terminal device 20 is used. The terminal device 20 is used by a second user U2 who is an assistant. The terminal device 20 is, for example, an information processing terminal such as a personal computer, a notebook computer, a tablet terminal, or a smartphone. The terminal device 20 includes a display device 201. Display device 201 is an example of a second display device. The terminal device 20 displays the captured image transmitted from the wearable device 10 on the display device 201. The captured image shows at least a portion of the visual field range R1 (see FIG. 6) of the first user U1. Specifically, the photographed image shows a space in which the devices DV1 to DV3 are arranged and the first user U1 working on the devices DV1 to DV3. By displaying such a captured image on the terminal device 20, the second user U2 can provide advice and monitor the work performed by the first user U1 from a remote location. In addition, for example, by saving captured images on the terminal device 20 or the wearable device 10, it is possible to verify after the fact whether the work procedure and work contents of the first user U1 were correct, or to investigate the cause of a problem when it occurs. You can do it.

A-2. Wearable device 10
FIG. 4 is a block diagram showing the configuration of the wearable device 10. Wearable device 10 includes a transmissive display device 101, a speaker 102, a microphone 103, a communication device 104, an imaging device 105, a storage device 106, a processing device 107, and a bus 108. At least a portion of the configuration shown in FIG. 4 is stored in frame F, for example. The transmissive display device 101, the speaker 102, the microphone 103, the communication device 104, the imaging device 105, the storage device 106, and the processing device 107 are interconnected by a bus 108 for communicating information. . The bus 108 may be configured using a single bus, or may be configured using different buses for each element such as a device.

The transmissive display device 101 includes left and right lenses L, a display panel, and optical members. The display panel and the optical member are housed in a frame F, for example. The display panel and the optical member may be provided in pairs on the left and right, corresponding to the left and right lenses L. The transmissive display device 101 displays a projection image corresponding to a virtual object on a display panel based on control from the processing device 107. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The optical member guides the light emitted from the display panel to the left and right lenses L.

The left and right lenses L each have a half mirror. The half mirrors of the left and right lenses L transmit the light representing the real space, thereby guiding the light representing the real space to the eyes of the first user U1. Further, the half mirrors of the left and right lenses L reflect the light indicating the virtual object guided by the optical member toward the eyes of the first user U1. The light in the real space that has passed through the half mirror and the light that indicates the virtual object reflected by the half mirror are incident on the eyes of the first user U1, so that the first user U1 can see the position of the virtual object in the real space. Perceive it as if you are doing it.

The speaker 102 plays back the audio data and outputs audio corresponding to the audio data. The audio data reproduced by the speaker 102 is, for example, audio data representing the utterance of the second user U2, which is collected by the microphone 204 of the terminal device 20. The microphone 103 picks up surrounding sounds and generates audio data. The voice picked up by the microphone 103 is, for example, the voice uttered by the first user U1 wearing the wearable device 10. The speaker 102 and microphone 103 are arranged in frame F, for example. Alternatively, the speaker 102 and the microphone 103 may be separate from the wearable device 10 without being included in the wearable device 10.

The communication device 104 includes a communication interface for communicating with other devices. The communication device 104 connects to the communication network N using wireless communication or wired communication, and communicates with the terminal device 20 via the communication network N.

The imaging device 105 includes, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system that includes at least one imaging lens. The imaging lens is provided, for example, on the bridge mentioned above. Therefore, the imaging device 105 images the outside world in the direction in which the first user U1's face is facing. The imaging optical system may include various optical elements such as a prism, or may include a zoom lens, a focus lens, or the like. The image sensor is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

The imaging device 105 images a subject and generates captured image data corresponding to the captured image (hereinafter referred to as "captured image"). As will be described later with reference to FIG. 6, in this embodiment, the imaging range R2 (see FIG. 6) of the imaging device 105 is the range that is visible through the lens L of the visual field R1 of the first user U1, i.e., the range that is visible through the lens L; It includes the display range R3 (see FIG. 6) of the mold display device 101. The correspondence between each pixel of the captured image captured by the imaging device 105 and each pixel of the transmissive display device 101 is calibrated in advance. Therefore, for example, taking an object A that appears in a captured image as an example, when the first user U1 views the real space through the transmissive display device 101, the range that the object A occupies in the display range R3 of the transmissive display device 101 is Known. Therefore, the first display control unit 111, which will be described later, can project a virtual object whose display position is determined based on the captured image onto the transmissive display device 101.

Captured image data generated by the imaging device 105 is transmitted to the terminal device 20 via the communication device 104. The imaging device 105 repeats imaging at predetermined imaging intervals, and transmits the generated captured image data to the terminal device 20 each time the imaging is performed. In this embodiment, it is assumed that the photographed image data is moving image data.

The storage device 106 is a recording medium that can be read by the processing device 107. Storage device 106 includes, for example, nonvolatile memory and volatile memory. Examples of nonvolatile memories include ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Memory). Programmable Read Only Memory). The volatile memory is, for example, RAM (Random Access Memory). Storage device 106 stores program PG1. Program PG1 is a program for operating wearable device 10.

The processing device 107 includes one or more CPUs (Central Processing Units). One or more CPUs are an example of one or more processors. A processor and a CPU are each an example of a computer.

The processing device 107 reads the program PG1 from the storage device 106. The processing device 107 functions as the first display control section 111 and the first communication control section 112 by executing the program PG1. At least one of the first display control unit 111 and the first communication control unit 112 is a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Circuit). Logic Device) and FPGA (Field Programmable Gate Array), etc. It may also be configured by a circuit. Details of the first display control section 111 and the first communication control section 112 will be described later.

A-3. Terminal device 20
FIG. 5 is a block diagram showing the configuration of the terminal device 20. As shown in FIG. The terminal device 20 includes a display device 201, an input device 202, a speaker 203, a microphone 204, a communication device 205, a storage device 206, a processing device 207, and a bus 208. The display device 201, the input device 202, the speaker 203, the microphone 204, the communication device 205, the storage device 206, and the processing device 207 are interconnected by a bus 208 for communicating information. The bus 208 may be configured using a single bus, or may be configured using different buses for each device.

The display device 201 is a display device (eg, various display panels such as a liquid crystal panel and an organic EL panel) that displays information to the outside. The input device 202 is an input device (eg, keyboard, mouse, switch, button, sensor, etc.) that accepts input from the outside. Note that the display device 201 and the input device 202 may have a configuration in which they are integrated (for example, a touch panel).

The speaker 203 plays back the audio data and outputs audio corresponding to the audio data. The audio data reproduced by the speaker 203 is, for example, audio data representing the voice uttered by the first user U1, which is picked up by the microphone 103 of the wearable device 10. The microphone 204 picks up surrounding sounds and generates audio data. The voice picked up by the microphone 204 is, for example, the voice uttered by the second user U2 who uses the terminal device 20. The speaker 203 and the microphone 204 may not be included in the terminal device 20 and may be separate from the terminal device 20.

The communication device 205 includes a communication interface for communicating with other devices. The communication device 104 connects to the communication network N using wireless communication or wired communication, and communicates with the wearable device 10 via the communication network N.

The storage device 206 is a recording medium that can be read by the processing device 207. Storage device 206 includes, for example, nonvolatile memory and volatile memory. Non-volatile memories are, for example, ROM, EPROM and EEPROM. Volatile memory is, for example, RAM. Storage device 206 stores program PG2. The program PG2 is a program for operating the terminal device 20.

The processing device 207 includes one or more CPUs. One or more CPUs are an example of one or more processors. A processor and a CPU are each an example of a computer.

The processing device 207 reads the program PG2 from the storage device 206. The processing device 207 functions as a second communication control section 211, a second display control section 212, and a reception section 213 by executing the program PG2. At least one of the second communication control section 211, the second display control section 212, and the reception section 213 may be configured by a circuit such as a DSP, an ASIC, a PLD, and an FPGA. Details of the second communication control section 211, second display control section 212, and reception section 213 will be described later.

A-4. Details of the processing device 107 and the processing device 207 The details of the processing device 107 and the processing device 207 will be described below. Prior to this, the relationship between the visual field range R1 of the first user U1, the imaging range R2 of the imaging device 105, and the display range R3 of the transmissive display device 101 will be described. FIG. 6 is an explanatory diagram schematically showing a range related to the visual field range R1 of the first user U1. Note that the ranges R1 to R3 shown in FIG. 6 are intended to relatively indicate the size of each range, and are different from the actual viewing angles. In FIG. 6, symbol E schematically indicates the eyes of the first user U1. Generally, it is known that the human visual field range, that is, the visual field range R1 of the first user U1, is about 200° in the horizontal direction (left-right direction) and about 130° in the vertical direction (up-down direction).

On the other hand, the imaging range R2 of the imaging device 105 is about 100° in the horizontal direction and 80° in the vertical direction when the imaging lens of the imaging device 105 is a wide-angle lens. Therefore, the imaging range R2 of the imaging device 105 is narrower than the visual field range R1 of the first user U1. Note that this is not the case, for example, when a special lens such as a fisheye lens is used as the imaging lens of the imaging device 105. Moreover, the starting point of the imaging range R2 is, strictly speaking, a position different from the eye E of the first user U1. However, as described above, the imaging lens of the imaging device 105 is provided on the bridge, and the starting point of the imaging range R2 and the eye E of the first user U1 can be considered to be at approximately the same position. Furthermore, since the wearable device 10 is attached to the head of the first user U1, the range of the real space that is captured in the captured image changes depending on the movement of the first user U1. Therefore, the captured image of the imaging device 105 captures the real space included in the visual field range R1 of the first user U1.

The display range R3 of the transmissive display device 101 is a range that is visible through the lens L of the visual field range R1 of the first user U1. As described above, the transmissive display device 101 allows the light of the real space that has passed through the half mirror provided on the lens L and the light representing the virtual object reflected by the half mirror to enter the user's eyes. , causes the first user U1 to perceive a virtual object. Therefore, in the real space, the range in which the virtual object can be displayed by the transmissive display device 101 is the range that the first user U1 visually recognizes through the lens L. In this embodiment, the display range R3 of the transmissive display device 101 is narrower than the viewing range R1 of the first user U1. It is also possible to make the display range R3 and the visual field range R1 of the first user U1 equal. Furthermore, in the present embodiment, the display range R3 of the transmissive display device 101 is narrower than the imaging range R2 of the imaging device 105, but as described above, the display range R3 of the transmissive display device 101 is set to a wider range. If so, it is also possible to make the display range R3 of the transmissive display device 101 larger than the imaging range R2 of the imaging device 105.

Next, the processing device 107 of the wearable device 10 will be explained. As described above, the processing device 107 functions as the first display control section 111 and the first communication control section 112. The first display control unit 111 controls the transmissive display device 101 located in front of the first user U1.

The first communication control unit 112 controls data transmission and reception by the communication device 104. The first communication control section 112 is an example of a transmitting section and a receiving section. The first communication control unit 112 transmits, to the terminal device 20, a captured image (more specifically, captured image data corresponding to the captured image) in which the real space included in the field of view of the first user U1 is captured, for example.

The processing device 207 of the terminal device 20 functions as a second communication control section 211, a second display control section 212, and a reception section 213. The second communication control unit 211 controls data transmission and reception by the communication device 205. The second communication control unit 211 receives a captured image (more specifically, captured image data corresponding to the captured image) from the wearable device 10, for example.

The second display control unit 212 controls the display device 201. For example, the second display control unit 212 displays the captured image on the display device 201. The receiving unit 213 receives a designation of a specific region RX in the captured image displayed on the display device 201. The specific area RX is an example of a first area. The specific region RX may be any region, but in this embodiment, it is a particularly important region of the captured image and is a region that is desired to be continuously included in the captured image. The receiving unit 213 receives the designation of the specific region RX, for example, by receiving drawing of a frame indicating the outer edge of the specific region RX on the captured image displayed on the display device 201. This frame is drawn using the input device 202, for example.

7A and 7B are diagrams showing an example of a captured image displayed on the display device 201. The captured image M1 shown in FIG. 7A shows an area including the devices DV1 and DV2 in the real space. For example, assume that while the first user U1 is working, the second user U2 needs to monitor the status of the device DV1 (for example, the display of the meter, the connection status of the connector, etc.). Therefore, the range including the device DV1 in the captured image M1 is designated as the specific region RX. Specifically, the reception unit 213 displays a region designation button B1 superimposed on the captured image M1. When the second user U2 presses the region designation button B1, the reception unit 213 enters the region reception mode in which designation of the specific region RX is accepted, and as shown in FIG. 7B, superimposes a cursor C resembling a pen on the captured image M1. Display. The second user U2 draws a frame W to surround the device DV1 by moving a cursor C using a mouse, which is an example of the input device 202, for example. When the second user U2 presses the OK button B2, the reception unit 213 accepts the area inside the frame W as the specific area RX. The reception unit 213 generates area information that specifies the specific area RX. The area information is, for example, position information (for example, coordinate values) of each point of the frame W on the captured image M1. The second communication control unit 211 transmits region information that specifies the position of the specific region RX in the captured image M1 to the wearable device 10.

When the wearable device 10 receives the area information from the terminal device 20, the first display control unit 111 transparently displays the first virtual object V1 indicating the range corresponding to the specific area RX in the real space based on the area information. It is displayed on the device 101. The first virtual object V1 includes, for example, a first virtual frame V1-1 and a message image V1-2. The first virtual frame V1-1 is an example of a frame and a first frame. The first virtual frame V1-1 indicates a position corresponding to the outer edge of the specific region RX in the real space. Furthermore, the message image V1-2 includes a message urging the first user U1 to maintain the state in which the first virtual frame V1-1 is included in the display range R3 of the transmissive display device 101. Message image V1-2 is an example of the first message. Note that only the first virtual frame V1-1 may be displayed as the first virtual object V1.

FIGS. 8A and 8B are diagrams showing an example of the display range R3 of the transmissive display device 101. As described above, the display range R3 of the transmissive display device 101 is the range that is visible through the lens L of the visual field range R1 of the first user U1. Furthermore, the display range R3 of the transmissive display device 101 is narrower than the imaging range R2 of the imaging device 105. In the example of FIGS. 8A and 8B, the display range R3 of the transmissive display device 101 includes the entire device DV1 and a part of the device DV2.

Until the second user U2 specifies the specific region RX, only the real space is visible in the display range R3, as shown in FIG. 8A. When the second user U2 specifies the specific area RX, a first virtual frame V1-1, which is an example of the first virtual object V1, is displayed on the lens L, as shown in FIG. 8B. The first virtual frame V1-1 is a frame having the same shape as the frame W drawn by the second user U2. The first virtual frame V1-1 indicates the outer edge of the range RY corresponding to the specific region RX in the captured image in the real space.

As described above, the correspondence between each pixel of the captured image captured by the imaging device 105 and each pixel of the transmissive display device 101 is calibrated in advance. Therefore, the first display control unit 111 can display the first virtual frame V1-1 corresponding to the frame W drawn on the captured image on the transmissive display device 101.

FIG. 9 is a diagram showing another example of the display range R3 of the transmissive display device 101. FIG. 8B shows a display range R3 when the first user U1 is located in front of the device DV1. FIG. 9 shows a display range R3 when the first user U1 is located diagonally horizontally with respect to the device DV1. As described above, the first virtual frame V1-1 is arranged at the outer edge of the range RY corresponding to the specific region RX of the captured image in the real space. Therefore, as shown in FIG. 9, when the first user U1 is viewing the device DV1 diagonally from the side, the first virtual frame V1-1 is also drawn as if he is viewing the device from the side diagonally.

Furthermore, as shown in FIG. 8B, the first display control unit 111 displays the message image V1-2 as the first virtual object V1. The message image V1-2 includes a message urging the first user U1 to adjust the posture so that the entire first virtual frame V1-1 falls within the display range R3. When the display range R3 includes the entire first virtual frame V1-1, the image captured by the imaging device 105 includes the specific region RX. On the other hand, if part or all of the first virtual frame V1-1 deviates from the display range R3, there is a possibility that part or all of the specific region RX is not included in the image captured by the imaging device 105. The first user U1 works while adjusting the orientation of his or her face so that the first virtual frame V1-1 always falls within the display range R3. Therefore, the specific region RX is always included in the image captured by the imaging device 105. In addition to displaying the first virtual object V1, the speaker 102 may output a voice that informs the first user U1 that the specific region RX has been set.

FIG. 10 is a diagram showing an example of a display when a part of the first virtual frame V1-1 is removed from the display range R3. For example, the first user U1's attention may be focused on the work and may not notice that the first virtual frame V1-1 has moved away from the display range R3. In the example of FIG. 10, the upper side of the rectangular first virtual frame V1-1 is outside the display range R3. In this way, when at least a part of the first virtual object V1 moves out of the display range R3 of the transparent display device 101, the first display control unit 111 further moves the second virtual object V2 to prompt the first user U1 to move. indicate.

In the example of FIG. 10, the second virtual object V2 includes a message image V2-1 and an arrow image V2-2. The message image V2-1 includes a message urging the first user U1 to turn his face upward so that the entire frame (first virtual frame V1-1) enters the lens L. The arrow image V2-2 indicates the direction in which the first user U1 should move his or her face. The arrow indicated by the arrow image V2-2 is an example of a symbol indicating the direction in which the first user should move. Note that it is not limited to both the message image V2-1 and the arrow image V2-2 being displayed as the second virtual object V2, but only one of them may be displayed. That is, the second virtual object V2 includes a message image V2-1 urging the first user U1 to move so that the first virtual frame V1-1 is included in the display range R3 of the transparent display device 101, and a message image V2-1 for the first user U1. At least one of the arrow image V2-2 indicating the direction in which the arrow should move is displayed.

Along with the second virtual object V2, a voice that calls attention to the first user U1 may be output from the speaker 102. The first user U1 who visually recognized the second virtual object V2 is predicted to move his or her face so that the entire first virtual frame V1-1 enters the lens L. Therefore, the imaging range of the imaging device 105 can be adjusted so that the entire specific region RX is captured in the captured image.

Note that, as shown in FIG. 6, in this embodiment, the imaging range R2 of the imaging device 105 is wider than the display range R3 of the transmissive display device 101. Therefore, even if a part of the first virtual frame V1-1 deviates from the display range R3, there is a possibility that the specific region RX will not be missing on the captured image. On the other hand, by displaying the second virtual object V2 at the time when a part of the first virtual frame V1-1 is removed from the display range R3, it is possible to prevent the specific region RX from being missing on the captured image. I can do it. Therefore, imaging of the entire specific region RX is performed more reliably.

Furthermore, although the entire first virtual frame V1-1 is within the display range R3, if the first user U1 is located far from the device DV1, the range of the specific region RX shown in the captured image becomes smaller. In this case, it becomes difficult for the second user U2 to visually recognize the specific area RX, and there is a possibility that the second user U2 cannot obtain necessary information. Therefore, when the ratio of the range RY corresponding to the specific area RX in the real space to the display range R3 becomes equal to or less than a predetermined value, the first display control unit 111 controls the first display control unit 111 to display a third Virtual object V3 is further displayed. The above predetermined value can also be said to be the proportion of the specific region RX in the captured image. The predetermined value may be determined by the second user U2 or the first user U1, or may be determined by the wearable device 10 (for example, the first display control unit 111).

FIG. 11 is a diagram showing an example of a display when the ratio of the range RY to the display range R3 is equal to or less than a predetermined value. The first display control unit 111 displays the first virtual object V1 and the third virtual object V3, as shown in FIG. 11, for example. The first virtual object V1 is a first virtual frame V1-1. The first virtual frame V1-1 indicates a position corresponding to the outer edge of the specific region RX in the real space. The first virtual frame V1-1 is displayed. The third virtual object V3 includes a second virtual frame V3-1 surrounding the first virtual frame V1-1, and a first user U1 such that the second virtual frame V3-1 is outside the display range R3 of the transparent display device 101. Message image V3-2 containing a message urging movement. The second virtual frame V3-1 is an example of a second frame. Message image V3-2 is an example of the second message.

The second virtual frame V3-1 is displayed at a location a predetermined distance away from the first virtual frame V1-1. The range outside the second virtual frame V3-1 is, so to speak, a range that must not be seen by the first user U1. The second virtual frame V3-1 is displayed, for example, in a different display mode (for example, a different color or a different line type) than the first virtual frame V1-1. The message image V3-2 displays a message urging the first user U1 to move (forward) until the dotted line frame (second virtual frame V3-1) is no longer visible.

Note that not all of the second virtual frame V3-1, message image V3-2, and arrow image V3-3 are displayed as the third virtual object, but any one or more of these may be displayed. Further, for example, instead of the second virtual frame V3-1, an image that fills in (masks) the outside of the second virtual frame V3-1 may be displayed. Furthermore, along with the display of the third virtual object V3, a sound may be output from the speaker 102 to alert the first user U1.

A-5. Summary of the Embodiments As described above, according to the embodiment, the display control system 1 transmits the captured image captured by the wearable device 10 to the terminal device 20, and in the terminal device 20, the specific region RX on the captured image Accepts specifications. In addition, the display control system 1 uses the transmissive display device 101 of the wearable device 10 to display a first virtual object V1 indicating a range corresponding to the specific region RX in the real space in the visual field R1 of the first user U1. indicate. Therefore, the first user U1 can grasp the range corresponding to the specific region RX in the real space, and can efficiently capture an image showing the range corresponding to the specific region RX in the real space.

The display control system 1 also displays, as the first virtual object V1, a first virtual frame V1-1 indicating a position corresponding to the outer edge of the specific region RX in the real space, and a transparent display of the first virtual frame V1-1. A message image V1-2 containing a message urging the first user U1 to maintain the state included in the display range R3 of the device 101 is displayed. Therefore, the first user U1 can intuitively grasp the position corresponding to the outer edge of the specific region RX in the real space, and can appropriately maintain his or her posture. Furthermore, by visually recognizing the message image V1-2, the first user U1 can understand that the first virtual frame V1-1 should be maintained within the display range R3. Therefore, for example, even when the first user U1 is not accustomed to using the wearable device 10, or when many virtual objects are displayed in the display range R3, the range corresponding to the specific region RX in the real space can be captured. Images can be easily captured.

Furthermore, when at least a portion of the first virtual frame V1-1 is out of the display range R3, the display control system 1 further displays a second virtual object V2 that prompts the first user U1 to move. Therefore, even when the first user U1 is concentrating on work, for example, he can notice that a part of the first virtual frame V1-1 is outside the display range R3.

The display control system 1 also displays, as the second virtual object V2, a message image V2-1 or a message image V2-1 containing a message urging the first user U1 to move so that the display range R3 includes the first virtual frame V1-1 At least one of the arrow image V2-2 indicating the direction in which the user U1 should move is displayed. Therefore, the first user U1 can specifically grasp the movement he/she should take, and can work efficiently.

The display control system 1 also provides a third virtual object that prompts the first user U1 to move when the ratio of the range RY corresponding to the specific region RX in the real space to the display range R3 becomes equal to or less than a predetermined value. Display more V3. Therefore, it is possible to prevent the specific region RX shown in the captured image from becoming too small, and to more efficiently support the work of the second user U2.

The display control system 1 also creates a second virtual frame V3-1 that surrounds the first virtual frame V1-1 as a third virtual object V3, and a first virtual frame V3-1 that surrounds the first virtual frame V1-1, and a first A message image V3-2 containing a message urging the user U1 to move is displayed. Therefore, the first user U1 can specifically grasp the movement he/she should take, and can work efficiently.

B: Modifications Modifications of the above embodiment are shown below. Two or more aspects arbitrarily selected from the following modified aspects may be combined as appropriate to the extent that they do not contradict each other.

B1: First Modification In the embodiment described above, the wearable device 10 was configured only with a glasses-type terminal device. The wearable device 10 is not limited to this, and may include a glasses-type terminal device and a portable terminal device such as a smartphone, a tablet terminal, or a notebook computer. In this case, the eyeglass-type terminal device and the portable terminal device are connected to each other. When the wearable device 10 includes a portable terminal device, the first display control section 111 and the first communication control section 112 may be partially or entirely controlled by the portable terminal device.

C: Others (1) Each function illustrated in FIG. 4 or 5 is realized by any combination of hardware and software. The method for realizing each function is not particularly limited. Each function may be realized using one physically or logically coupled device, or may be realized using two or more physically or logically separated devices directly or indirectly (e.g., wired, It may also be realized using devices configured by connecting (e.g., wirelessly). Each function may be realized by combining software with the one device or the plurality of devices.

(2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

(3) In each of the embodiment and the first modification, the storage device 106 and the storage device 206 include an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (for example, a compact disk, a digital versatile disc, a Blu-ray disc), a smart card, a flash memory (e.g. card, stick, key drive), a floppy disc, a magnetic strip, etc. good. The program may also be transmitted from a network via a telecommunications line.

(4) Each of the embodiment and the first modification is based on LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile com communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile commu nication system (xG) (x is an integer or a decimal number, for example), FRA (Future Radio Access), NR (new Radio), New radio access (NX ), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-F i (registered trademark)), IEEE 802. 16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other appropriate systems, and systems that are extended, modified, created, and defined based on these. It may be applied to at least one next generation system. Furthermore, a combination of a plurality of systems may be applied (for example, a combination of at least one of LTE and LTE-A and 5G).

(5) The processing procedures, sequences, flowcharts, etc. illustrated in each of the embodiment and the first modification may be rearranged in order as long as there is no contradiction. For example, the methods described herein present elements of the various steps in an exemplary order and are not limited to the particular order presented.

(6) In each of the embodiment and the first modification, input/output information, etc. may be stored in a specific location (for example, memory) or may be managed using a management table. Information etc. to be input/output may be overwritten, updated, or additionally written. The output information etc. may be deleted. The input information etc. may be transmitted to other devices.

(7) In each of the embodiment and the first modification, the determination may be made based on a value represented by 1 bit (0 or 1), or a truth value (Boolean: true or false). The determination may be performed based on a comparison of numerical values (for example, a comparison with a predetermined value).

(8) The programs illustrated in each of the embodiments and the first modification are instructions, instruction sets, Should be broadly construed to mean code, code segment, program code, subprogram, software module, application, software application, software package, routine, subroutine, object, executable file, thread of execution, procedure or function, etc. . Additionally, software, instructions, etc. may be transmitted and received via a transmission medium. For example, if the software uses wired technology (such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL)) and/or wireless technology (such as infrared, microwave) to from a remote source, these wired and/or wireless technologies are included within the definition of a transmission medium.

(9) The information and the like described in each of the embodiments and the first variation may be represented using any of a variety of different techniques. For example, data, information, etc. that may be referred to throughout the above description may be represented in voltages, currents, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms explained in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

(10) In each of the embodiment and the first modification, the terms "system" and "network" are used interchangeably.

(11) In each of the embodiment and the first modification, the terminal device 20 may be a mobile station. A mobile station is defined by a person skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

(12) A mobile station may be called a transmitting device, a receiving device, a communication device, or the like. The mobile station may be a device mounted on a mobile body, or the mobile body itself. A moving body means a movable object. The moving speed of the moving object is arbitrary. The moving body can be stopped. Examples of moving objects include vehicles, transportation vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, carts, rickshaws, ships and other watercraft, Including, but not limited to, airplanes, rockets, artificial satellites, drones (registered trademarks), multicopters, quadcopters, balloons, and objects mounted thereon. The mobile body may be a mobile body that autonomously travels based on the operation command. The moving object may be a vehicle (for example, a car, an airplane, etc.), an unmanned moving object (for example, a drone, a self-driving car, etc.), or a robot (manned or unmanned). There may be. Mobile stations also include devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

(13) In each of the embodiment and the first modification, the term "determining" may include a wide variety of operations. "Decision" includes, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring. y) (e.g. table , searching in a database or other data structure), asserting something as a "decision," and the like. In addition, "determination" refers to receiving (e.g., receiving information), transmitting (e.g., transmitting information), input, output, accessing ( For example, it may include accessing data in memory) and regarding it as a "judgment" or "decision." Furthermore, "determining" may include determining that the process of resolving, selecting, choosing, establishing, comparing, etc. has been "determined." In other words, "determining" may include considering that some action has been "determined." Moreover, "determination" may be read as "assuming," "expecting," "considering," or the like.

(14) In each of the embodiments and the first variant, the term "connected", or any variation thereof, refers to any connection or coupling, direct or indirect, between two or more elements. can include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The bonds or connections between elements may be physical, logical, or a combination thereof. For example, "connection" may be replaced with "access." As used in this disclosure, two elements may include one or more wires, cables, and/or printed electrical connections, as well as in the radio frequency domain, as some non-limiting and non-inclusive examples. , electromagnetic energy having wavelengths in the microwave and optical (both visible and invisible) ranges, and the like.

(15) In each of the embodiments and the first modification, the statement "based on" does not mean "based only on" unless specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

(16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the amount or order of those elements. These designations may be used herein as a convenient way of distinguishing between two or more elements. Thus, reference to a first and second element does not imply that only two elements may be employed or that the first element must precede the second element in any way.

(17) In each of the embodiments and the first modification, when "include", "including", and variations thereof are used in this specification or the claims, these The term, like the term "comprising," is intended to be inclusive. Furthermore, the term "or" as used in this specification or in the claims is not intended to be exclusive or.

(18) Throughout this application, when articles are added by translation, such as a, an, and the in English, this disclosure does not include that the nouns following these articles are plural. good.

(19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented as modifications and variations without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description herein is intended to be illustrative and does not have any limiting meaning to the present invention. Further, a plurality of aspects selected from the aspects illustrated in this specification may be combined.

DESCRIPTION OF SYMBOLS 1... Display control system, 10... Wearable device, 20... Terminal device, 101... Transmissive display device, 104... Communication device, 105... Imaging device, 106... Storage device, 107... Processing device, 111... First display control unit , 112...first communication control unit, 201...display device, 202...input device, 205...communication device, 206...storage device, 207...processing device, 211...second communication control unit, 212...second display control unit, 213...Reception section, F...Frame, L...Lens, N...Communication network.

Claims (7)

A display control system comprising a wearable device and a terminal device,
The wearable device includes:
a first display control unit that controls a transmissive first display device located in front of the first user;
a first communication control unit that transmits a captured image of a real space included in the visual field of the first user to the terminal device,
The terminal device is
a second communication control unit that receives the captured image;
a second display control unit that displays the captured image on a second display device;
a reception unit that receives from a second user a designation of a first region in the captured image displayed on the second display device;
The second communication control unit transmits area information specifying a range of the first area in the captured image to the wearable device,
The first display control unit displays a first virtual object indicating a range corresponding to the first area in the real space on the first display device based on the area information.
Display control system. The first virtual object includes a frame and a first message,
The frame indicates a position corresponding to an outer edge of the first area in the real space,
The message prompts the first user to maintain the frame within a display range of the first display device.
The display control system according to claim 1. A range of the real space that is reflected in the captured image changes depending on the movement of the first user,
The first display control unit further displays a second virtual object that prompts the first user to move when at least a portion of the first virtual object is out of a display range of the first display device.
The display control system according to claim 1. The second virtual object includes a message prompting the first user to move so that the first virtual object is included in the display range of the first display device, and a symbol indicating a direction in which the first user should move. including at least one of
The display control system according to claim 3. A range of the real space that is reflected in the captured image changes depending on the movement of the first user,
The first display control unit controls the movement of the first user when a proportion of the range corresponding to the first area in the real space to the display range of the first display device becomes equal to or less than a predetermined value. further displaying a third virtual object that prompts
The display control system according to claim 1. The first virtual object is a first frame,
The first frame indicates a position corresponding to an outer edge of the first area in the real space,
The third virtual object includes a second frame surrounding the first frame, and a second message prompting the first user to move so that the second frame is out of the display range of the first display device.
The display control system according to claim 5. a first display control unit that controls a transmissive first display device located in front of the first user;
a transmitter that transmits a captured image of a real space included in the visual field of the first user to a terminal device;
a receiving unit that receives area information specifying a range of a first area specified in the captured image,
The first display control unit displays a first virtual object indicating a range corresponding to the first area in the real space on the first display device based on the area information.
Wearable device.

Images