WO2023176165A1

WO2023176165A1 - Information processing device

Info

Publication number: WO2023176165A1
Application number: PCT/JP2023/002432
Authority: WO
Inventors: 桃子阿部
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2022-03-17
Filing date: 2023-01-26
Publication date: 2023-09-21

Abstract

According to the present invention, a display control unit causes a virtual object to be displayed in a display region of AR glasses. An image acquiring unit acquires a first captured image including a range of real space viewed through the display region by a user wearing the AR glasses. An object detecting unit detects a prescribed object appearing in the first captured image. The AR glasses are configured to reduce light in a prescribed wavelength band among light incident on the display region. The display control unit causes a partial image representing a part of the first captured image corresponding to the prescribed object to be displayed in a position in the display region in which the user views the prescribed object.

Description

information processing equipment

The present invention relates to an information processing device.

Conventionally, products have been developed for the purpose of adjusting the brightness around a user or reducing the transmittance of light of a specific wavelength such as blue light. For example, Patent Document 1 listed below discloses a surface protection film in which four layers are laminated in this order: a protective layer made of polyurethane, a transparent base film, a blue light cutting layer, and an adhesive layer.

Patent No. 6992063

In recent years, XR glasses that apply XR technology represented by AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) have become popular. For example, when a surface protection film such as that disclosed in Patent Document 1 is attached to XR glasses, the user's entire visual field becomes dark, and objects within the visual field may become difficult to see.

An object of the present invention is to control the light incident on the XR glasses without blocking the user's field of view.

An information processing device according to one aspect of the present invention includes a display control unit that displays a virtual object in a display area of a see-through head-mounted display, and a virtual object that a user wearing the see-through head-mounted display visually recognizes through the display area. The see-through head-mounted display includes an image acquisition unit that acquires a captured image including a spatial range from an imaging device, and an object detection unit that detects a predetermined object appearing in the captured image. The display control unit is configured to reduce light in a predetermined wavelength band among the incident light, and the display control unit displays a partial image representing a portion of the predetermined object in the captured image when the user The predetermined object is displayed at a position where it can be visually recognized.

According to one aspect of the present invention, it is possible to control the light incident on the XR glasses without blocking the user's field of view.

FIG. 1 is a diagram showing the configuration of a display system 10 according to a first embodiment. 3 is an explanatory diagram showing the appearance of AR glasses 30. FIG. 3 is a block diagram showing the configuration of AR glasses 30. FIG. 2 is a block diagram showing the configuration of a terminal device 20. FIG. FIG. 3 is a diagram showing an example of a first captured image PA of a real space R; 3 is a diagram illustrating an example of a real space R visually recognized by a user wearing AR glasses 30. FIG. FIG. 3 is a diagram illustrating an example of a real space R that is visually recognized by a user when a partial image PD is displayed. It is a block diagram showing the composition of AR glasses 30 in a second embodiment. FIG. 2 is a block diagram showing the configuration of a terminal device 20 in a second embodiment. FIG. 3 is a diagram illustrating an example of a real space R that is visually recognized by a user when a partial image PT is displayed. FIG. 3 is a diagram illustrating an example of a real space R that is visually recognized by a user when a partial image PT is displayed. FIG. 6 is a diagram illustrating an example of a real space R that is visually recognized by a user when an outer edge image PE is displayed.

A. First Embodiment Hereinafter, the configuration of a display system 10 including an information processing apparatus according to a first embodiment of the present invention will be described.

A-1. System Configuration FIG. 1 is a diagram showing the configuration of a display system 10 according to the first embodiment. The display system 10 includes a terminal device 20 and AR glasses 30. The terminal device 20 is an example of an information processing device. The AR glasses 30 are an example of a see-through head mounted display. The terminal device 20 and the AR glasses 30 are connected to be able to communicate with each other. The terminal device 20 is preferably a mobile terminal device such as a smartphone or a tablet.

The display system 10 is a system that uses AR technology to present various information to a user wearing AR glasses 30, which will be described later. Here, AR technology is, for example, in a device such as a see-through head-mounted display, by displaying a virtual object superimposed on the coordinate space of the real world, so that the user wearing the device can receive information shown by the virtual object. This is a technology that makes it visible. Note that in addition to displaying the virtual object, the AR glasses 30 may also output other types of information, such as audio information. The virtual object is, for example, a still image, a moving image, a 3DCG model, text, or the like.

In this embodiment, a case will be described in which the virtual objects are partial images PD and PT, which will be described later. It's okay. In this case, the terminal device 20 is communicably connected to the virtual object server via a wide area communication network (not shown). The terminal device 20 receives virtual object data corresponding to a virtual object from the virtual object server. The terminal device 20 displays a virtual object on the AR glasses 30 using the virtual object data.

A-2. AR glasses 30
The AR glasses 30 are glasses-shaped see-through head-mounted displays that are worn on the user's head. The AR glasses 30 display virtual objects on the

lenses

310A and 310B under the control of the terminal device 20. Note that as the see-through head-mounted display, for example, a goggle-shaped see-through head-mounted display having the same functions as the AR glasses 30 may be used.

FIG. 2 is an explanatory diagram showing the appearance of the AR glasses 30. In the AR glasses 30,

temples

301 and 302, a bridge 303,

frames

304 and 305,

rims

306 and 307,

lenses

310A and 310B, and an imaging lens LEN are visible from the outside.

Bridge 303 connects

rims

306 and 307. An imaging lens LEN that constitutes the first imaging device 324 shown in FIG. 3 is arranged on the bridge 303.

The frame 304 is provided with a display panel for the left eye and an optical member for the left eye. The frame 305 is provided with a display panel for the right eye and an optical member for the right eye. The display panel for the left eye and the display panel for the right eye are, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye and the display panel for the right eye display images corresponding to virtual objects, for example, based on control from the terminal device 20 described later. The left eye optical member guides the light emitted from the left eye display panel to the lens 310A. The optical member for the right eye guides the light emitted from the display panel for the right eye to the lens 310B. Further, the

frames

304 and 305 are provided with speakers 322, which will be described later.

The rim 306 holds the lens 310A. Rim 307 holds lens 310B.

Each of the

lenses

310A and 310B has a half mirror. The half mirrors included in the

lenses

310A and 310B transmit the light representing the real space R, thereby guiding the light representing the real space R to the user's eyes. Further, the half mirrors included in the

lenses

310A and 310B reflect the light representing the virtual object guided by the optical member toward the user's eyes. The light of the real space R that has passed through the half mirror and the light that is reflected by the half mirror that indicates the virtual object are superimposed and incident on the user's eyes, allowing the user to see the position of the virtual object on the real space R. Perceive it as if you are doing it. That is, the

lenses

310A and 310B function as a transmissive display placed in front of the user's eyes.

Lenses

310A and 310B are display areas of AR glasses 30.

Furthermore, in this embodiment, a blue light cut filter F is attached to the outer surfaces of the

lenses

310A and 310B of the AR glasses 30. The outer surface of the

lenses

310A, 310B is the surface opposite to the surface facing the eyes of the user wearing the AR glasses 30, of the two surfaces of the

planar lenses

310A, 310B. The blue light cut filter F is a filter that reflects or absorbs blue light having a short wavelength of 380 nm or more and 500 nm or less among visible light (wavelengths of 380 nm or more and 780 nm or less). That is, the AR glasses 30 are configured to include the blue light cut filter F so as to reduce light in a predetermined wavelength band among the light incident on the

lenses

310A and 310B. The predetermined wavelength band is from 380 nm to 500 nm. Note that the blue light cut filter F may cut not only blue light but also ultraviolet light having a wavelength of less than 380 nm.

By using the blue light cut filter F, the amount of blue light that enters the eyes of the user wearing the AR glasses 30 can be reduced, and the burden on the user such as eye strain can be reduced. Note that in this embodiment, the blue light cut filter F cuts out blue light emitted from the display screens of electronic devices placed around the user wearing the AR glasses 30 and blue light contained in natural light incident from windows, etc. It is attached to

lenses

310A and 310B for the purpose of reducing light.

Note that instead of the blue light cut filter F, for example, an ND (Neutral Density) filter that reduces incident light to the

lenses

310A and 310B over the entire band may be used. By using an ND filter, the user can take a rest or concentrate on work even in bright surroundings. Also, for example, lenses 310A, 310
As B, a blue light cut lens may be used. In this case, the

lenses

310A and 310B themselves serve as light control members.

FIG. 3 is a block diagram showing the configuration of the AR glasses 30. The AR glasses 30 include the above-described

temples

301, 302, bridges 303, frames 304, 305,

rims

306, 307,

lenses

310A, 310B, and imaging lens LEN, as well as a projection device 321, a speaker 322, and the like. , a communication device 323, a first imaging device 324, a storage device 327, a processing device 328, and a bus 329.

Each configuration shown in FIG. 3 is stored in

frames

304 and 305, for example. The projection device 321, the speaker 322, the communication device 323, the first imaging device 324, the storage device 327, and the processing device 328 are interconnected by a bus 329 for communicating information. The bus 329 may be configured using a single bus, or may be configured using different buses for each element such as a device.

The projection device 321 includes

lenses

310A and 310B, a display panel for the left eye, an optical member for the left eye, a display panel for the right eye, and an optical member for the right eye. As described above, the projection device 321 displays the image corresponding to the virtual object on the left eye display panel and the right eye display panel based on the control from the terminal device 20. Light emitted from the display panel for the left eye is guided to the lens 310A by the optical member for the left eye. The light emitted from the display panel for the right eye is guided to the lens 310B by the optical member for the right eye. The user visually recognizes images corresponding to virtual objects reflected in

lenses

310A and 310B.

A speaker 322 is located in each of the

frames

304 and 305. The speaker 322 may not be located in each of the

frames

304 and 305, but may be located in, for example, one of the

frames

304 and 305, at least one of the

temples

301 and 302, or at least one of the bridge 303. The speaker 322 is controlled by the terminal device 20 directly or via the processing device 328 of the AR glasses 30. The audio output by the speaker 322 is, for example, audio output accompanying a virtual object. The speaker 322 may not be included in the AR glasses 30 and may be separate from the AR glasses 30.

The communication device 323 includes a communication interface for communicating with other devices. The communication device 323 communicates with the terminal device 20 using wireless communication or wired communication. In this embodiment, the communication device 323 communicates with the communication device 203 (see FIG. 4) of the terminal device 20 using short-range wireless communication such as Bluetooth (registered trademark).

The first imaging device 324 includes, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens LEN (see FIG. 2). For example, the imaging optical system may include various optical elements such as a prism, or may include a zoom lens, a focus lens, or the like. The image sensor is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

The first imaging device 324 images the subject and outputs first captured image information indicating the captured image (hereinafter referred to as "first captured image PA"). In this embodiment, the imaging range of the first imaging device 324 matches or includes the user's viewing range. Therefore, the first captured image PA includes the range of the real space R that is visually recognized by the user wearing the AR glasses 30 through the

lenses

310A and 310B. The correspondence relationship between each pixel of the first captured image PA captured by the first imaging device 324 and each pixel of the see-through display realized by the

lenses

310A and 310B is calibrated in advance. That is, it is known which position of the

lenses

310A and 310B the object appearing in the first captured image PA occupies when viewed from the user wearing the AR glasses 30. Therefore, the display control unit 214, which will be described later, can project a virtual object whose display position is determined based on the first captured image PA onto the

lenses

310A and 310B.

The first captured image PA generated by the first imaging device 324 is transmitted to the terminal device 20 via the communication device 323 as first captured image information. The first imaging device 324 repeats imaging at predetermined imaging intervals, and transmits the generated first captured image information to the terminal device 20 each time it captures an image.

The storage device 327 is a recording medium that can be read by the processing device 328. The storage device 327 includes, for example, nonvolatile memory and volatile memory. Examples of nonvolatile memories include ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Memory). Programmable Read Only Memory). The volatile memory is, for example, RAM (Random Access Memory). The storage device 327 stores program PG1. The program PG1 is a program for operating the AR glasses 30.

The processing device 328 includes one or more CPUs (Central Processing Units). One or more CPUs are an example of one or more processors. A processor and a CPU are each an example of a computer.

The processing device 328 reads the program PG1 from the storage device 327. The processing device 328 functions as the operation control unit 330 by executing the program PG1. The operation control unit 330 includes a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), and a PLD (Programmable Logic). It may be configured by a circuit such as an FPGA (Field Programmable Gate Array) or an FPGA (Field Programmable Gate Array).

The operation control unit 330 controls the operation of the AR glasses 30. For example, the operation control unit 330 provides the projection device 321 with an image display control signal that the communication device 323 receives from the terminal device 20 . The projection device 321 displays the image indicated by the image display control signal on the

lenses

310A and 310B.

A-3. Terminal device 20
FIG. 4 is a block diagram showing the configuration of the terminal device 20. As shown in FIG. Terminal device 20 includes a touch panel 201 , a communication device 203 , a storage device 205 , a processing device 206 , and a bus 207 . The touch panel 201, the communication device 203, the storage device 205, and the processing device 206 are interconnected by a bus 207 for communicating information. The bus 207 may be configured using a single bus, or may be configured using different buses for each device.

The touch panel 201 displays various information to the user and detects the user's touch operations. The touch panel 201 serves as both an input device and an output device. For example, the touch panel 201 is configured by bonding a touch sensor unit capable of detecting a touch operation between various display panels such as a liquid crystal display panel or an organic EL display panel and a cover glass. For example, when the user's finger is in contact with the touch panel 201, the touch panel 201 periodically detects the contact position of the user's finger on the touch panel 201, and sends touch information indicating the detected contact position to the processing device 206. do.

The communication device 203 includes a communication interface for communicating with other devices. The communication device 203 communicates with the AR glasses 30 using wireless communication or wired communication. In this embodiment, the communication device 203 communicates with the communication device 323 (see FIG. 3) using the same short-range wireless communication as the communication device 323 of the AR glasses 30.

Furthermore, the communication device 203 may communicate with the virtual object server using wireless communication or wired communication and may receive virtual object data. In this case, the communication device 203 includes an interface connectable to a wide area communication network (not shown).

The storage device 205 is a recording medium that can be read by the processing device 206. Storage device 205 includes, for example, nonvolatile memory and volatile memory. Non-volatile memories are, for example, ROM, EPROM and EEPROM. Volatile memory is, for example, RAM. Storage device 205 stores program PG2. The program PG2 is a program for operating the terminal device 20.

The processing device 206 includes one or more CPUs. One or more CPUs are an example of one or more processors. A processor and a CPU are each an example of a computer.

The processing device 206 reads the program PG2 from the storage device 205. The processing device 206 functions as an image acquisition section 210, an object detection section 212, and a display control section 214 by executing the program PG2. At least one of the image acquisition section 210, the object detection section 212, and the display control section 214 may be configured by a circuit such as a DSP, an ASIC, a PLD, and an FPGA.

The image acquisition unit 210 acquires from the first imaging device 324 a captured image that includes the range of the real space R that the user wearing the AR glasses 30 views through the

lenses

310A and 310B. In the present embodiment, the image acquisition unit 210 acquires first captured image information corresponding to the first captured image PA captured by the first imaging device 324 of the AR glasses 30 via the communication device 203.

The object detection unit 212 detects a predetermined object appearing in the first captured image PA. In this embodiment, the "object" is not limited to an inanimate object, but may be a living object (person, plant). In the first embodiment, the predetermined object is a display screen of an electronic device. The object detection unit 212 detects the display screen of the electronic device from the first captured image PA. The display screen of an electronic device may have a main function of displaying information to a user, such as a display screen of a television, personal computer, smartphone, tablet, or the like. Further, the display screen of the electronic device may have a main function of receiving settings for the electronic device from the user, such as a display screen of a printer, a portable music player, a cooking appliance, or the like.

The object detection unit 212 detects the display screen of the electronic device using, for example, image analysis. In addition, the object detection unit 212 detects the brightness or contrast ratio of the display screen reflected in the first captured image PA, and if the brightness or contrast ratio is below a predetermined value, the display screen (or the electronic device itself) is used. It may be determined that the state is not present and not detected.

The display control unit 214 causes the

lenses

310A and 310B of the AR glasses 30 to display the virtual object. For example, the display control unit 214 controls the projection device 321 of the AR glasses 30 using virtual object data acquired from a virtual object server (not shown) to display a virtual object corresponding to the virtual object data.

In the present embodiment, the display control unit 214 displays a partial image PD representing a portion of a predetermined object in the first captured image PA at a position of the

lenses

310A and 310B where the user visually recognizes the predetermined object. In the first embodiment, the partial image PD is a portion of the first captured image PA that represents the display screen of the electronic device. The display control unit 214 displays the partial images PD (for example, PD1 to PD3) at a position of the

lenses

310A and 310B where the user views the display screen.

The reason for performing such a display is to improve the visibility of the display screen from the user's perspective. As mentioned above, the blue light cut filter F is attached to the AR glasses 30. Therefore, the visibility of the display screen is reduced compared to the state where the AR glasses 30 are not worn. On the other hand, in many cases, the display screen is arranged to display information required by the user, and it is desired that the user can easily understand the content displayed on the display screen. Therefore, the display control unit 214 generates a partial image PD representing the display screen of the electronic device out of the first captured image PA captured by the first imaging device 324. Then, the display control unit 214 displays the partial image PD superimposed on the position where the user views the display screen. Therefore, compared to the case where the display screen is viewed through the AR glasses 30, the visibility of the display screen is improved. That is, according to the display system 10, the user can easily understand the displayed content on the display screen while reducing blue light.

FIG. 5 is a diagram showing an example of the first captured image PA of the real space R. The first captured image PA shown in FIG. 5 is captured so as to include the range of the real space R that the user wearing the AR glasses 30 views through the

lenses

310A and 310B. The real space R shown in the first captured image PA is indoors, and a television OB1, a notebook computer OB2, a smartphone OB3, a lighting fixture OB4, a window OB5, and the like are arranged. Of the first captured image PA, a portion where the display screen of the television OB1 is shown is a partial image PD1, a portion where the display screen of the notebook computer OB2 is shown is a partial image PD2, and a portion where the display screen of the smartphone OB3 is shown is a partial image PD3. Hereinafter, the television OB1, notebook computer OB2, and smartphone OB3 may be referred to as electronic devices OB1 to OB3. The color of the real space R reflected in the first captured image PA is almost the same as the color perceived by a user who is not wearing the AR glasses 30.

FIG. 6 is a diagram showing an example of the real space R visually recognized by the user wearing the AR glasses 30. FIG. 6 shows the real space R that the user views when the partial image PD is not displayed. The real space R shown in FIG. 6 is visually recognized by the user through the blue light cut filter F. Therefore, the amount of blue light that enters the user's eyes is reduced. On the other hand, compared to FIG. 5, the real space R appears darker overall.

FIG. 7 is a diagram showing an example of the real space R that the user visually recognizes when the partial image PD is displayed. In FIG. 7, a partial image PD1 is displayed superimposed on the display screen of the television OB1, a partial image PD2 is displayed superimposed on the display screen of the notebook computer OB2, and a partial image PD3 is displayed superimposed on the display screen of the smartphone OB3. Is displayed. As described above, the color tone of the real space R reflected in the first captured image PA is almost the same as the color tone perceived by a user who is not wearing the AR glasses 30. Therefore, the display screens of the television OB1, notebook computer OB2, and smartphone OB3 are viewed in the same way as when the user is not wearing the AR glasses 30.

Each of the partial images PD1 to PD3 is an image of the same size as the display screen of each electronic device OB1 to OB3 that is viewed by the user. That is, the display control unit 214 displays a partial image PD having the same size as the display area viewed by the user. Therefore, the user can view the partial image PD without feeling uncomfortable, and the user's fatigue caused by using the AR glasses 30 is reduced.

Furthermore, the partial image PD may be displayed in a so-called night mode color tone. In night mode, the colors of the image are changed to warmer colors. Therefore, the amount of blue light that enters the user's eyes is reduced, and the use of the display screen is prevented from interfering with subsequent sleep. In the night mode, the color temperature of the white point of the display screen is set lower than normal. That is, the display control unit 214 may set the color temperature of the white point of the partial images PD displayed on the

lenses

310A and 310B to a predetermined temperature or lower. The predetermined temperature, ie, the color temperature of the white point in the night mode, may be specifiable by the user. For example, when the color temperature of the white point in normal mode is 6500K (equivalent to daylight color), the color temperature of the white point in night mode may be less than 5500K (equivalent to daylight white) and more than 2700K (equivalent to light bulb color). By displaying the partial image PD in the color of the night mode, blue light entering the user's eyes is reduced, further reducing the burden on the user such as eye strain.

A-4. Summary of the First Embodiment As described above, the display system 10 according to the first embodiment is configured to reduce light in a predetermined wavelength band among the light incident on the display area of the AR glasses 30. Furthermore, the display system 10 displays a partial image PD representing a predetermined object in the real space R at a position in the display area of the AR glasses 30 where the user visually recognizes the predetermined object. Therefore, the light incident on the AR glasses 30 can be controlled without blocking the user's field of view.

Furthermore, the display system 10 according to the first embodiment is configured to reduce light of 380 nm or more and 500 nm or less, which is considered blue light, among the light incident on the display area of the AR glasses 30. Furthermore, the display system 10 displays a partial image PD representing the display screen of the electronic device. Therefore, blue light generated from the display screen of the electronic device is reduced, and the user can easily understand the content displayed on the display screen. Note that the predetermined object is not limited to the display screen, but may be an object designated by the user.

Furthermore, the display system 10 according to the first embodiment displays the partial image PD in night mode. That is, the display system 10 sets the color temperature of the white point of the partial image PD reflected in the display area of the AR glasses 30 to a predetermined temperature or lower. Therefore, blue light in images reflected on the

lenses

310A and 310B is reduced, and the burden on the user, including eye strain, can be further reduced.

Furthermore, the display system 10 according to the first embodiment displays partial images PD1 to PD3 that have the same size as the display screens of the electronic devices OB1 to OB3 that are viewed by the user. Therefore, the partial image PD is displayed without any discomfort, and the user's fatigue caused by using the AR glasses 30 is reduced.

B. Second Embodiment A second embodiment of the present invention will be described below. In the following description, in order to simplify the description, the same reference numerals are used for the same components as in the first embodiment, and the description of their functions may be omitted. In addition, in the following description, in order to simplify the description, the differences between the second embodiment and the first embodiment will be mainly described.

FIG. 8 is a block diagram showing the configuration of AR glasses 30 in the second embodiment. In the second embodiment, the AR glasses 30 include a second imaging device 331 in addition to the configuration of the AR glasses 30 of the first embodiment shown in FIG. The second imaging device 331 has imaging lenses (not shown) on the surfaces of the

rims

306 and 307 that face the user's eyes when the user wears the AR glasses 30 . The second imaging device 331 then captures an image including the user's eyes. The image captured by the second imaging device 331 is referred to as a second captured image. The second captured image generated by the second imaging device 331 is transmitted to the terminal device 20 via the communication device 323 as second captured image information. The second imaging device 331 repeats imaging at predetermined imaging intervals, and transmits the generated second captured image information to the terminal device 20 each time it captures an image.

FIG. 9 is a block diagram showing the configuration of the terminal device 20 in the second embodiment. In the second embodiment, the processing device 206 of the terminal device 20 functions as an image acquisition section 210, an object detection section 212, a display control section 214, and a tracking section 216 by executing the program PG2.

As in the first embodiment, the image acquisition unit 210 acquires a first captured image PA of an area corresponding to the user's visual field range, and also acquires a second captured image that captures an area including the user's eyes. In the present embodiment, the image acquisition unit 210 obtains first captured image information corresponding to the first captured image PA captured by the first imaging device 324 of the AR glasses 30 and the first captured image information corresponding to the first captured image PA captured by the second imaging device 331 of the AR glasses 30. second captured image information corresponding to the captured second captured image is acquired.

The tracking unit 216 tracks the user's eye movements. In this embodiment, the tracking unit 216 analyzes the movement of the user's eyes in the second captured image to estimate which position in the first captured image PA the user's line of sight is directed to. Specifically, the tracking unit 216 uses the inner corner of the eye as a reference point, for example, and estimates the position toward which the user's line of sight is directed based on the position of the iris relative to the inner corner of the eye. For example, if the iris of the right eye is far from the inner corner of the eye, it can be estimated that the user's line of sight is facing to the right, and if the inner corner of the right eye and the iris are close, it can be estimated that the user's line of sight is facing to the left. . Note that the tracking unit 216 may track the movement of the user's eyes using, for example, a corneal reflex method.

The object detection unit 212 detects the first object appearing in the captured image based on the user's eye movements tracked by the tracking unit 216. The first object is an object that the user gazes at. That is, in the second embodiment, the predetermined object is the first object that the user gazes at. The object detection unit 212 detects an object appearing in the first captured image PA using, for example, image analysis. Then, the object detection unit 212 detects an object that overlaps with the position of the user's line of sight as a first object. Note that, for example, if the user's line of sight continues toward a specific object for a predetermined period of time (for example, 2 seconds, etc.), the object detection unit 212 may detect the specific object as the first object. good.

In the second embodiment, the display control unit 214 generates partial images PT (for example, PT1, PT2) showing the first object in the first captured image PA. The display control unit 214 displays the partial image PT in the display area of the AR glasses 30 at a position where the user visually recognizes the first object. As mentioned above, the blue light cut filter F is attached to the AR glasses 30. Therefore, the visibility of the first object is reduced compared to a state where the user is not wearing the AR glasses 30. Therefore, the display control unit 214 generates a partial image PT representing the first object out of the first captured image PA captured by the first imaging device 324. Then, the display control unit 214 displays the partial image PT at the position where the user visually recognizes the first object. By displaying the partial image PT, the visibility of the first object is improved compared to the case where the first object is viewed through the AR glasses 30. That is, according to the display system 10, the user can easily see the first object while reducing blue light.

FIGS. 10 and 11 are diagrams showing an example of the real space R that the user visually recognizes when the partial image PT is displayed. In FIGS. 10 and 11, the user's gaze point (the point toward which the user's line of sight is directed) is indicated by the symbol TP. FIG. 10 shows a case where the first object is a window OB5. In this case, partial image PT1 is displayed superimposed on window OB5. Partial image PT1 is a portion of first captured image PA shown in FIG. 5 that represents window OB5. As described above, the color tone of the real space R reflected in the first captured image PA is almost the same as the color tone perceived by a user who is not wearing the AR glasses 30. By displaying the partial image PT1, the window OB5 is visually recognized in the same way as when the user is not wearing the AR glasses 30.

FIG. 11 shows a case where the first object is the display screen of smartphone OB3. In this case, partial image PT2 is displayed superimposed on smartphone OB3. Partial image PT2 is a portion of the first captured image PA shown in FIG. 5 that represents the display screen of smartphone OB3. By displaying the partial image PT2, the display screen of the smartphone OB3 is visually recognized in the same way as when the user is not wearing the AR glasses 30.

Note that the display control unit 214 may display the partial image PT when the first object is a specific object. The specific object is, for example, a display screen of an electronic device. In this case, the display control unit 214 does not display the partial image PT1 when the first object is the window OB5 as shown in FIG. 10, and does not display the partial image PT1 when the first object is the smartphone OB3 as shown in FIG. The partial image PT2 is displayed. By displaying the partial image PT only when the first object is a specific object, it is possible to display only the object that the user particularly wants to improve visibility as the partial image PT, and reduce the display frequency of the partial image PT.

Note that also in the second embodiment, the partial image PT may be displayed in the same size as the first object visually recognized by the user. Further, in the second embodiment as well, the partial image PT may be displayed in a color tone in night mode.

As described above, the display system 10 according to the second embodiment displays the partial image PT that includes the first object that the user is gazing at. By displaying the partial image PT in which the first object is captured, blue light generated around the user is reduced, and visibility of the first object by the user is improved.

C: Modification Examples Modifications of the above embodiment are shown below. Two or more aspects arbitrarily selected from the following modified aspects may be combined as appropriate to the extent that they do not contradict each other.

C1: First Modified Example In the first embodiment and the second embodiment, the display control unit 214 displays partial images PD and PT in which the entire predetermined object is captured by the user using the

lenses

310A and 310B. Displayed in a visible position. For example, in the first embodiment, the display control unit 214 displays the partial image PD, which shows the entire display screen of the electronic device, in a superimposed manner at the position where the user visually recognizes the display screen. The display control unit 214 is not limited to this, and the display control unit 214 may display an outer edge image PE (for example, PE1 to PE3) indicating the outer edge of a predetermined object out of the partial images PD and PT when the user selects the outer edge image PE (for example, PE1 to PE3) of the It may be displayed at a position where the outer edge is visible.

FIG. 12 is a diagram showing an example of the real space R that the user visually recognizes when the outer edge image PE is displayed. In FIG. 12, an outer edge image PE1 is displayed at a position where the user visually recognizes the outer edge of the display screen of the television OB1. Further, an outer edge image PE2 is displayed at a position where the user visually recognizes the outer edge of the display screen of the notebook computer OB2. Further, an outer edge image PE3 is displayed at a position where the user visually recognizes the outer edge of the display screen of the smartphone OB3. By displaying the outer edge image PE, the user can grasp the position of the display screen of the electronic device, and it becomes easier to understand the display contents of the display screen than when the outer edge image PE is not displayed. Furthermore, the outer edge image PE has a smaller image area than the partial image PD. Therefore, when displaying the outer edge image PE, there is less strain on the user's eyes compared to when displaying the partial image PD.

Note that in FIG. 12, the outer edge image PE is an image showing a predetermined range inside from the outer edge of the display screen of the electronic device. For example, the outer edge image PE may be an image showing a predetermined range outside the outer edge of the display screen of the electronic device, or a predetermined range including the inside and outside of the outer edge of the display screen of the electronic device. It may be an image.

According to the first modification, the burden on the user's eyes is further reduced, and the visibility of a predetermined object is improved.

C2: Second Modification In the first embodiment and the second embodiment, the partial images PD and PT are displayed regardless of the distance between the user and the predetermined object. The present invention is not limited to this, and whether or not to display the partial images PD and PT may be switched depending on the distance between the user and a predetermined object. Specifically, when the distance between the user and the predetermined object is less than a predetermined distance, partial images PD and PT showing the predetermined object are displayed, and when the distance between the user and the predetermined object is greater than or equal to the predetermined distance. In this case, the partial images PD and PT indicating the predetermined object may not be displayed. Generally, when the distance between the user and a predetermined object is large, the visibility of the predetermined object is low even if the user is not wearing the AR glasses 30. Therefore, even if the partial images PD and PT are displayed, there is little influence on the visibility of a predetermined object. That is, by displaying the partial images PD, PT only when the distance between the user and the predetermined object is less than the predetermined distance, the effectiveness of displaying the partial images PD, PT can be improved.

In this case, the object detection unit 212 detects the distance between the user and the predetermined object. For example, when the size of the predetermined object is known, the object detection unit 212 detects the user (more specifically, the AR glasses 30 worn by the user) based on the size of the predetermined object appearing in the first captured image PA. The distance between the object and the predetermined object may be estimated. Further, if the AR glasses 30 are equipped with a LiDAR (Light Detection and Ranging) sensor, the object detection unit 212 may detect the distance between the user and a predetermined object using the detection value of the sensor. .

The display control unit 214 displays the partial images PD and PT on the AR glasses 30 when the distance between the user and the predetermined object is less than a predetermined distance. Further, the display control unit 214 does not display the partial images PD and PT on the AR glasses 30 when the distance between the user and the predetermined object is a predetermined distance or more.

The predetermined distance may be changed depending on the size of the object, for example. For example, the display screen of the television OB1 shown in FIG. 5 and the like is larger than the display screen of the smartphone OB3. Therefore, the user may view the display screen of the television OB1 from a position 3 meters away from the television OB1, for example. On the other hand, even if the display screen of the smartphone OB3 is viewed from a position 3 meters away, it is considered that the content reflected on the display screen is hardly discernible. Therefore, the display control unit 214 may increase the predetermined distance as the size of the predetermined object becomes larger.

According to the second modification, the partial images PD and PT are displayed only when the distance between the user and the predetermined object is less than the predetermined distance, so the effectiveness of displaying the partial images PD and PT is improved.

C3: Third Modified Example In the first embodiment and the second embodiment, the partial images PD and PT were displayed in the same size as the predetermined object viewed from the user. For example, in the first embodiment, partial images PD1 to PD3 having the same size as the display screen in the user's visual field are displayed. The present invention is not limited to this, and, for example, the partial images PD and PT may be larger than the predetermined object viewed from the user. By displaying the partial images PD, PT larger than the predetermined object seen by the user, the visibility of the partial images PD, PT can be improved, and the user's convenience can be improved. Furthermore, the user may be able to set the ratio between the size of a predetermined object in the user's visual field and the display size of the partial images PD and PT.

C4: Fourth Modification In the first embodiment and the second embodiment, the blue light cut filter F was attached to the outer surfaces of the

lenses

310A and 310B of the AR glasses 30. However, the present invention is not limited to this, and the blue light cut filter F may be attached to the inner surfaces of the

lenses

310A and 310B of the AR glasses 30. On the other hand, if the blue light cut filter F is attached to the inner surfaces of the

lenses

310A and 310B of the AR glasses 30, the images (virtual objects including partial images PD and PT) displayed by the projection device 321 become difficult to see. Since the image displayed by the projection device 321 can reduce blue light by displaying it in night mode, for example, it is preferable to attach the blue light cut filter F to the outer surface of the

lenses

310A and 310B as described above. It is.

C5: Fifth Modification In the first embodiment and the second embodiment, the AR glasses 30 and the terminal device 20 were separate bodies. The present invention is not limited to this, and for example, the AR glasses 30 may have the functions of the terminal device 20. That is, the processing device 328 of the AR glasses 30 may function as the image acquisition section 210, the object detection section 212, the display control section 214, and the tracking section 216.

According to the fifth modification, the terminal device 20 is not required, which is advantageous in simplifying the system configuration.

D: Others (1) Each function illustrated in FIGS. 3, 4, 8, and 9 is realized by any combination of hardware and software. The method for realizing each function is not particularly limited. Each function may be realized using one physically or logically coupled device, or may be realized using two or more physically or logically separated devices directly or indirectly (for example, wired, It may also be realized using devices configured by connecting (e.g., wirelessly). Each function may be realized by combining software with the one device or the plurality of devices.

(2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

(3) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the storage device 205 and the storage device 327 include an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, flexible disks, magneto-optical disks (e.g. compact disks, digital versatile disks, Blu-ray disks), smart cards, flash memories (e.g. cards, sticks, key drives), floppy disks, It may be constituted by at least one of magnetic strips and the like. The program may also be transmitted from a network via a telecommunications line.

(4) Each of the first embodiment, the second embodiment, and the first to fifth modifications are LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation on mobile communication system (6G), xth generation mobile communication system (xG) (x is an integer or a decimal, for example), FRA (Future Rad io Access) , NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 ( Systems that utilize Wi-Fi (registered trademark), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other appropriate systems, and these It may be applied to at least one of the next-generation systems extended, modified, created, or defined based on the invention. Furthermore, a combination of a plurality of systems may be applied (for example, a combination of at least one of LTE and LTE-A and 5G).

(5) The processing procedures, sequences, flowcharts, etc. illustrated in each of the first embodiment, the second embodiment, and the first to fifth modifications may be rearranged in order as long as there is no contradiction. For example, the methods described herein present elements of the various steps in an exemplary order and are not limited to the particular order presented.

(6) In each of the first embodiment, the second embodiment, and the first to fifth modifications, input/output information, etc. may be stored in a specific location (for example, memory) or managed. It may also be managed using a table. Information etc. to be input/output may be overwritten, updated, or additionally written. The output information etc. may be deleted. The input information etc. may be transmitted to other devices.

(7) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the determination may be made based on the value (0 or 1) represented by 1 bit. However, it may be performed based on a truth value (Boolean: true or false), or may be performed based on a comparison of numerical values (for example, a comparison with a predetermined value).

(8) The programs illustrated in each of the first embodiment, the second embodiment, and the first to fifth modifications are called software, firmware, middleware, microcode, or hardware description language, or are called other names. an instruction, set of instructions, code, code segment, program code, subprogram, software module, application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, or function, whether referred to by should be broadly interpreted to mean, etc. Additionally, software, instructions, etc. may be sent and received via a transmission medium. For example, if the software uses wired technologies (such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL)) and/or wireless technologies (such as infrared, microwave) to from a remote source, these wired and/or wireless technologies are included within the definition of a transmission medium.

(9) The information described in each of the first embodiment, second embodiment, and first to fifth modifications may be represented using any of various different techniques. For example, data, information, etc. that may be referred to throughout the above description may be represented in voltages, currents, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms explained in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

(10) In each of the first embodiment, second embodiment, and first to fifth modifications, the terms "system" and "network" are used interchangeably.

(11) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the terminal device 20 may be a mobile station. A mobile station is defined by a person skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

(12) A mobile station may be called a transmitting device, a receiving device, a communication device, or the like. The mobile station may be a device mounted on a mobile body, or the mobile body itself. A moving body means a movable object. The moving speed of the moving object is arbitrary. The moving body can be stopped. Examples of moving objects include vehicles, transportation vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, carts, rickshaws, ships and other watercraft, Including, but not limited to, airplanes, rockets, artificial satellites, drones (registered trademarks), multicopters, quadcopters, balloons, and objects mounted thereon. The mobile body may be a mobile body that autonomously travels based on the operation command. The moving object may be a vehicle (for example, a car, an airplane, etc.), an unmanned moving object (for example, a drone, a self-driving car, etc.), or a robot (manned or unmanned). There may be. Mobile stations also include devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

(13) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the term "determining" or "determining" encompasses a wide variety of operations. There are cases. "Decision" includes, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring. y) (e.g. table , searching in a database or other data structure), asserting something as a "decision," and the like. In addition, "determination" refers to receiving (e.g., receiving information), transmitting (e.g., transmitting information), input, output, accessing ( For example, it may include accessing data in memory) and regarding it as a "judgment" or "decision." Furthermore, "determining" may include determining that the process of resolving, selecting, choosing, establishing, comparing, etc. has been "determined." In other words, "determining" may include considering that some action has been "determined." Moreover, "determination" may be read as "assuming," "expecting," "considering," or the like.

(14) In each of the first embodiment, the second embodiment, and the first to fifth variations, the term "connected" or any variation thereof refers to two or more elements. refers to any connection or coupling, direct or indirect, between two elements that are "connected" or "coupled" to each other, and can include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The bonds or connections between elements may be physical, logical, or a combination thereof. For example, "connection" may be replaced with "access." As used in this disclosure, two elements may include one or more wires, cables, and/or printed electrical connections, as well as in the radio frequency domain, as some non-limiting and non-inclusive examples. , electromagnetic energy having wavelengths in the microwave and optical (both visible and non-visible) ranges, and the like.

(15) In each of the first embodiment, the second embodiment, and the first to fifth modifications, the statement "based on" does not mean "based only on" unless otherwise specified. It doesn't mean anything. In other words, the phrase "based on" means both "based only on" and "based at least on."

(16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the amount or order of those elements. These designations may be used herein as a convenient way of distinguishing between two or more elements. Thus, reference to a first and second element does not imply that only two elements may be employed or that the first element must precede the second element in any way.

(17) In each of the first embodiment, the second embodiment, and the first modification to the fifth modification, "include", "including", and variations thereof may be used in the present specification or When used in the claims, these terms, like the term "comprising", are intended to be inclusive. Furthermore, the term "or" as used in this specification or in the claims is not intended to be exclusive or.

(18) Throughout this application, when articles are added by translation, such as a, an, and the in English, this disclosure does not include that the nouns following these articles are plural. good.

(19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented as modifications and variations without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description herein is intended to be illustrative and does not have any limiting meaning to the present invention. Further, a plurality of aspects selected from the aspects illustrated in this specification may be combined.

10... Display system, 20... Terminal device, 30... AR glasses, 201... Touch panel, 203... Communication device, 205... Storage device, 206... Processing device, 207... Bus, 210... Image acquisition section, 212... Object detection section, 214... Display control unit, 216... Tracking unit, 321... Projection device, 322... Speaker, 323... Communication device, 324... First imaging device, 327... Storage device, 328... Processing device, 329... Bus, 330... Operation control 331...Second imaging device, F...Blue light cut filter, LEN...Imaging lens, R...Real space.

Claims

a display control unit that displays a virtual object in a display area of a see-through head-mounted display;
an image acquisition unit that acquires from an imaging device a captured image including a range of real space that is viewed through the display area by a user wearing the see-through head-mounted display;
an object detection unit that detects a predetermined object appearing in the captured image,
The see-through head mounted display is configured to reduce light in a predetermined wavelength band among the light incident on the display area,
The display control unit displays a partial image representing a portion of the predetermined object in the captured image at a position in the display area where the user visually recognizes the predetermined object.
Information processing device.
The predetermined object is a display screen of an electronic device,
The predetermined wavelength band is from 380 nm to 500 nm,
The object detection unit detects the display screen from the captured image,
The partial image represents the display screen of the captured image,
The display control unit displays the partial image at a position in the display area where the user views the display screen.
The information processing device according to claim 1.
The predetermined object is a first object that the user gazes at,
further comprising a tracking unit that tracks eye movements of the user,
The object detection unit detects the first object from the captured image based on the user's eye movement tracked by the tracking unit,
The partial image represents the first object in the captured image,
The display control unit displays the partial image in the display area at a position where the user visually recognizes the first object.
The information processing device according to claim 1.
The display control unit sets a color temperature of a white point of the partial image reflected in the display area to a predetermined temperature or less.
The information processing device according to claim 1.
The partial image is an image of the same size as the predetermined object visually recognized by the user,
the display control unit displays the partial image in the display area;
The information processing device according to claim 1.
The display control unit displays an outer edge image indicating an outer edge of the predetermined object among the partial images at a position in the display area where the user visually recognizes the outer edge of the predetermined object.
The information processing device according to claim 1.
The object detection unit detects a distance between the user and the predetermined object,
The display control unit displays the partial image in the display area when a distance between the user and the predetermined object is less than a predetermined distance.
The information processing device according to claim 1.