JP2023044464A - Picture display system, device for displaying picture, and method for displaying picture - Google Patents

Abstract

To provide a technique that allows an accurate overlap of a real space and a virtual space without much delay of follow-up of a picture to motions of a user.SOLUTION: A picture display system according to one preferred aspect of the present invention includes a feature point position recognition unit, a coordinate processing unit, and a picture display processing unit. The feature point position recognition unit recognizes the position of a feature point of a target on the basis of an image taken by an imaging unit. The coordinate processing unit determines a reference coordinate system on the basis of the position of the feature point, detects a motion of the imaging unit on the basis of sensor information shown by the sensor unit, and corrects the display position of a picture on the basis of the motion of the imaging unit. The picture display unit displays a picture in the corrected display position of a picture when displaying a picture on the basis of the reference coordinate system.SELECTED DRAWING: Figure 1

Description

The present invention relates to technology for displaying images, and more particularly to technology for superimposing images in virtual space on real space.

2. Description of the Related Art Video display devices and video display systems, such as head-mounted displays (HMDs), are known that present information to users by superimposing video in virtual space on real space. In factories, etc., there are cases where work is performed while viewing content such as the work process, but it is sometimes difficult to place a display or the like near the work target. In such cases, if a video display device such as a head-mounted display is used, the worker can perform the work while referring to the work instructions displayed on the video display device, thereby improving work efficiency. .

Japanese Patent Application Laid-Open No. 2002-200002 describes that, in a head-mounted display device, a virtual space is arranged at a position that is easy for the user to view, and an image that the user wants to see is comfortably selected and displayed.

Patent Document 2 describes a configuration for calculating the three-dimensional position of a feature point included in a captured image based on an image captured by a camera.

Japanese Unexamined Patent Application Publication No. 2020-123260 Japanese Patent Application Laid-Open No. 2008-304269

When a user works while looking at contents such as work instructions, the positional relationship between the real space and the virtual space is important. If the positional relationship between the real space and the virtual space changes when the user rotates or moves the head, the user may feel uncomfortable or may receive an incorrect work instruction.

In the head-mounted display device described in Patent Document 1, the differential amount of rotation of the head with respect to the waist of the user is calculated, and the display content displayed in the virtual space is changed according to the differential amount of rotation of the head. So, the positional relationship between the real space and the virtual space is matched. However, since the display position of the content is determined based on the movement of the user, the content may drift over time even if the head does not rotate.

Further, Patent Document 2 discloses an information processing device that performs self-position estimation and environment map creation by SLAM (Simultaneous Localization and Mapping). If the real space is recognized and the positional relationship between the real space and the virtual space is aligned, there is a problem that the image change of the image display device cannot follow the rotation of the user's head, in other words, a delay occurs.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to develop a technology that can superimpose the real space and the virtual space with high positional accuracy with little delay in following the motion of the user. to provide.

A preferred aspect of the present invention is a video display system comprising a feature point position recognition unit, a coordinate processing unit, and a video display processing unit, wherein the feature point position recognition unit recognizes an image captured by an imaging unit. The coordinate processing unit determines a reference coordinate system based on the position of the feature point, and determines the movement of the imaging unit based on the sensor information indicated by the sensor unit. and corrects the display position of the image based on the movement of the imaging unit, and the image display processing unit displays the image at the corrected display position of the image when displaying the image based on the reference coordinate system. It is a video display system characterized by displaying.

Another preferred aspect of the present invention is a video display device for displaying video, comprising: an imaging unit that captures an image of an object and acquires a captured image; a sensor unit that detects movement of the imaging unit; a coordinate processing unit; an image display processing unit; The coordinate processing unit determines a reference coordinate system based on the position of the feature point, generates correction information for the position of the image displayed by the image display unit based on the movement of the imaging unit, and The image display processing unit determines a display position of the image based on the reference coordinate system and the correction information, and the image display unit displays the image at the determined display position. It is a display device.

Another preferred aspect of the present invention is a video display method for displaying desired content, comprising: a first step of recognizing the position of a feature point of an object in an image captured by a camera; a second step of determining a reference coordinate system; a third step of detecting movement of the camera; and determining a display position of the content based on the movement of the camera when displaying the content based on the reference coordinate system. and a fourth step of correcting the image display method.

According to the present invention, it is possible to provide a technology capable of superimposing the real space and the virtual space with high positional accuracy with little delay in following the motion of the user. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a block diagram showing an example of functional blocks of a video display device according to an embodiment; FIG. 1 is a block diagram showing an example of a hardware configuration of a video display device according to an embodiment; FIG. 1 is a schematic diagram showing an example of a video display device of an embodiment and a user. FIG. 1 is a perspective view showing an example of a video display device according to an embodiment; FIG. 4A and 4B are schematic diagrams showing an example of a relationship between a physical space and a coordinate system according to the first embodiment; FIG. 4 is a flowchart showing an example of a flowchart relating to content display according to the first embodiment; FIG. FIG. 10 is a flowchart showing another example of a flowchart relating to content display of the first embodiment; FIG. FIG. 4 is a schematic diagram showing an example of feature points according to an embodiment; FIG. 4 is a schematic diagram showing an example of a method of determining display content by a video display processing unit according to the embodiment; FIG. 4 is a schematic diagram showing another example of a method of determining display content by the video display processing unit according to the embodiment; FIG. 4 is a schematic diagram showing an example of positions of display content displayed by the video display device according to the embodiment; FIG. 10 is a schematic diagram showing an example of a feature point and a reference coordinate system according to Example 2; FIG. 11 is a flowchart showing an example of a flowchart relating to content display according to the second embodiment; FIG. FIG. 11 is a block diagram showing an example of functional blocks of a video display device according to a third embodiment; FIG. 11 is a perspective view showing an example of a video display device according to Example 3; FIG. 11 is a flow chart showing an example of a flow chart leading to determination of a reference coordinate system according to the third embodiment; FIG. FIG. 11 is a schematic diagram showing an example of how markers are superimposed on the physical space according to the third embodiment; The block diagram of the remote work support system which is 4th Example which utilized the image display apparatus of Example. FIG. 2 is a block diagram showing an example of an overview of a video information transmission system of a remote work support system; FIG. 11 is a schematic diagram showing an example of a remote work support system according to a fourth embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, the following description is for describing one embodiment of the present invention, and does not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each or all of these elements are replaced with equivalents, and these embodiments are also included in the scope of the present invention.

In the configurations of the embodiments described below, the same reference numerals may be used in common for the same parts or parts having similar functions in different drawings, and redundant description may be omitted.

When there are a plurality of elements having the same or similar functions, they may be described with the same reference numerals and different suffixes. However, if there is no need to distinguish between multiple elements, the subscripts may be omitted.

Notations such as “first”, “second”, “third” in this specification etc. are attached to identify the constituent elements, and do not necessarily limit the number, order, or content thereof isn't it. Also, numbers for identifying components are used for each context, and numbers used in one context do not necessarily indicate the same configuration in other contexts. Also, it does not preclude a component identified by a certain number from having the function of a component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings, etc. may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings and the like.

All publications, patents and patent applications cited herein are hereby incorporated by reference into this description.

Elements presented herein in the singular shall include the plural unless the context clearly dictates otherwise.

A representative example of the embodiment is as follows. That is, the video display system includes an imaging section, a sensor section, a feature point position recognition section, a coordinate processing section, and a video display processing section. The feature point position recognition unit recognizes the position of the feature point of the object based on the image captured by the imaging unit, the coordinate processing unit determines a reference coordinate system based on the position of the feature point, Based on the information indicated by the sensor unit, a correction coordinate system or correction coordinates is determined by correcting the relative displacement of the imaging unit with respect to the reference coordinate system. The video display processing unit has a configuration for displaying content at a predetermined position in the corrected coordinate system, or displaying content at a corrected predetermined position in the reference coordinate system. Here, relative displacement is a concept that includes both positional displacement and angular displacement.

In this embodiment, the image display device 100 is a device having a function of displaying images, such as a head-mounted display, a head-up display, a smart phone, a tablet, or the like.

FIG. 1 is a diagram showing an example of functional blocks of a video display device 100 of this embodiment. The image display device 100 has an imaging unit 101 , a sensor unit 102 , a feature point position recognition unit 103 , a coordinate processing unit 104 , an image display processing unit 105 , an image display unit 106 and a controller 107 .

The image capturing unit 101 captures an image of the surrounding environment of the image display device 100 . The sensor unit 102 detects motion of the video display device 100 . Here, the concept of motion includes both position displacement and angle change (rotation). The feature point position recognition unit 103 recognizes the positions of feature points in the image captured by the imaging unit 101 . The coordinate processing unit 104 processes the coordinate system of the video displayed by the video display device 100 . The video display processing unit 105 performs display processing of the video displayed by the video display device 100 . The image display unit 106 displays images. The controller 107 centrally controls the entire image display device 100 . Details of each part will be described later.

FIG. 2 is a diagram showing a configuration example of the hardware configuration of the video display device 100 of this embodiment. The image display device 100 has a camera 111 , a sensor 112 , a display 113 , a CPU (Central Processing Unit) 114 , a RAM (Random Access Memory) 115 and a storage medium 116 .

In one configuration example shown in FIG. The configuration of the camera 111 may be a known configuration described in Patent Document 2 or the like.

Further, a sensor 112 is provided as the sensor unit 102 . The sensor 112 detects the movement of the camera 111, for example, the movement of the user's head. Also, the video display unit 106 has a display 113 . The configuration of the sensor unit 102 and the display 113 may be a known configuration described in Patent Document 1 or the like.

The display method of the display 113 may be a transmissive display that can receive visual information from the outside world, or a non-transmissive display that displays only video information. A transmissive display will be described. Two displays 113 may be provided for both eyes, or one may be provided for one eye.

CPU 114 executes programs stored in storage medium 116 or RAM 115 . Specifically, the CPU 114 executes the program to realize the functions of the controller 107 of the image display device 100, the feature point position recognition unit 103, the coordinate processing unit 104, the image display processing unit 105, and the like.

The storage medium 116 is a medium for storing programs executed by the CPU 114 and various parameters and data necessary for the execution.

The RAM 115 is a storage medium for storing images to be displayed on the display 113 and various information. The RAM 115 also functions as a temporary storage area for programs and data used by the CPU 114 . The image display device 100 may be configured to have a plurality of CPUs 114, RAMs 115, and storage media 116, respectively.

Note that the hardware configuration of the video display device 100 is not limited to the configuration shown in FIG. For example, the CPU 114 , the storage medium 116 and the RAM 115 may be provided separately from the image display device 100 . In that case, the image display device 100 may be implemented using a general-purpose computer (for example, a server computer, a personal computer, a smart phone, etc.).

For example, the camera 111, the sensor 112, and the display 113 are mounted on a wearable head-mounted display worn by the user, the CPU 114, the RAM 115, and the storage medium 116 are configured in a remote server, and the head-mounted display and the server are connected via a network. You can connect and configure.

Also, a plurality of computers can be connected via a network so that each computer can share the function of each section of the image display device 100 . On the other hand, one or more of the functions of the image display device 100 can also be realized using dedicated hardware.

FIG. 3 is a diagram showing an example of the video display device 100 and the user 700 of this embodiment. The image display device 100 shown in FIG. 3 is a head-mounted display (also called smart glasses) that can be used by a user 700 wearing it on his/her head.

FIG. 4 is a diagram showing an example of a mounting form of the video display device 100 of this embodiment. The image display device 100 includes a camera 111 and a sensor 112, and a gyro sensor is used as the sensor 112 in this example. The gyro sensor can detect movement of the image display device 100 .

The display 113 is built in the lens portion of the spectacles. In the case of a transmissive display, the user 700 can see the outside world through the display 113 and at the same time can see the image projected on the display 113 by the image display unit 106 . The camera 111 captures an image in front of the user, and the camera image substantially overlaps the external world visually recognized by the user. Next, the operation of the image display device 100 of this embodiment will be described in detail.

FIG. 5 is a diagram showing an example of the relationship between the physical space and the coordinate system in this embodiment. A detailed description will be given later.

FIG. 6 is a diagram showing an example of a flowchart relating to content display executed by the video display device 100 of this embodiment.

First, the video display device 100 acquires a camera image (S101). The controller 107 of the video display device 100 sends an imaging command to the imaging unit 101 , the imaging unit 101 takes an image according to the command, and the imaging unit 101 sends the captured image to the controller 107 .

As shown in FIGS. 3 and 4, the camera 111 included in the video display device 100 is configured to capture an image in front of the user 700 when the user 700 wears the video display device 100 on the head. It is attached to the video display device 100 . Therefore, by acquiring the image captured by the camera 111, the image display device 100 can acquire the situation in the real space in front of the user 700 as an image. That is, in the image captured by the camera 111, an object 131 located in front of the user in the physical space is captured (Fig. 5(a)). As the object 131, for example, a distribution board (an operation panel thereof) operated by the user 700 is assumed.

Next, the image display device 100 recognizes the positions of feature points (S102). The controller 107 of the video display device 100 sends the captured image received from the imaging unit 101 and the command to the feature point position recognition unit 103, and the feature point position recognition unit 103 recognizes the position of the feature point 132 from the captured image, and obtains the recognition result. is sent to the controller 107 (FIG. 5(a)). Details of the feature point 132 will be described later, but the feature point 132 is a characteristic point in the captured image received by the feature point position recognition unit 103 from the controller 107 . Also, the position of the feature point 132 indicates at which position in the captured image the feature point 132 is located.

Next, the image display device 100 determines a reference coordinate system (S103). The controller 107 of the image display device 100 sends the position of the feature points and the command received from the feature point position recognition unit 103 to the coordinate processing unit 104, and the coordinate processing unit 104 determines the reference coordinate system based on the positions of the feature points. , transmits the coordinate system information to the controller 107 .

In FIG. 5A, with the position of the feature point 132 as a reference, the x-axis 133 is set downward on the page, and the y-axis 134 is set rightward on the page. However, this embodiment is not limited to this, and a position shifted by a predetermined distance in a predetermined direction from the position of the feature point 132 may be set as the origin of the x-axis 133 and the y-axis 134, or a coordinate system in which the two axes are not orthogonal. , or the polar coordinate system. In the following description, as shown in FIG. 5A, the position of the feature point 132 is used as a reference, the x-axis 133 is set in the downward direction on the paper surface, and the y-axis 134 is set in the right direction on the paper surface.

In addition, the coordinate processing unit 104 has in advance information on the positional relationship between the image captured by the imaging unit 101 and the image displayed by the image display unit 106 . That is, based on the angle of view, direction, and rotation angle of the image captured by the image capturing unit 101 and the angle of view, direction, and rotation angle of the image displayed by the image display unit 106, a predetermined position within the image captured by the image capturing unit 101 is determined. and which direction corresponds to which position and direction in the image displayed by the image display unit 106 is provided in advance. As a result, the coordinates in the captured image and the coordinates in the video can be mutually converted. Information regarding such a positional relationship can be determined in advance based on the specifications of the imaging unit 101 and the image display unit 106 . Information for transforming the first coordinate system in the captured image and the second coordinate system in the video may be referred to as coordinate transformation information for convenience.

Based on the information about the positional relationship, the coordinate processing unit 104 uses the coordinate transformation information to transform the first reference coordinate system set in the captured image into the second reference coordinate system in the video displayed by the video display unit 106. Convert to The coordinate processing unit 104 transmits information regarding the second reference coordinate system in the image displayed by the image display unit 106 to the controller 107 . Specifically, for example, it is information about the origin and direction of the second reference coordinate system in the image displayed by the image display unit 106 .

In the above description, the coordinate processing unit 104 sets the first reference coordinate system in the captured image and then converts it to the second reference coordinate system in the image displayed by the image display unit 106. This embodiment is not limited to this. The coordinate processing unit 104 uses the coordinate conversion information to calculate which position in the image displayed by the image display unit 106 corresponds to the position of the feature point 132, and based on this position, the image display unit 106 A second reference reference coordinate system may be set within the image to be displayed. Thereby, it is possible to reduce the processing in the coordinate processing unit 104 . Recognition of the position of the feature point 132 and determination of the reference coordinate system do not require spatial recognition of the entire physical space using the entire captured image, so high-speed position recognition and determination of the reference coordinate system are possible. .

As described above, the first reference coordinate system set in the captured image is converted into the second reference coordinate system in the video displayed by the video display unit 106 . By positioning the content (object) to be displayed in the captured image based on the converted coordinate system, the positional relationship between the object and the content in the captured image can be determined. As a result, it is possible to determine the positional relationship between the object in the real space and the content in the virtual space. However, this assumes that the field of view of the camera 111 is fixed.

Next, the image display device 100 repeats the judgment as to whether or not the predetermined time has elapsed until the predetermined time has elapsed (S104). In other words, it waits until the predetermined time elapses. Specifically, for example, the controller 107 measures the elapsed time after receiving the coordinate system information from the coordinate processing unit 104, and determines at predetermined intervals whether the elapsed time exceeds a predetermined time. and repeats until the elapsed time exceeds a predetermined time.

Alternatively, the controller 107 has a loop counter, determines whether the loop counter exceeds a predetermined value, increases the value of the loop counter by a predetermined value each time the determination is made, and the loop counter reaches the predetermined value. Repeat until the value of is exceeded.

As another method, the controller 107 obtains the time, determines at predetermined intervals whether or not the time exceeds the predetermined time after the predetermined time, and determines whether the obtained time is the predetermined time. It is repeated until the time when the predetermined time has passed from is exceeded.

In general, the camera 111 moves as the user's head and body move over time. At the time of step S103 when the first and second reference coordinate systems are determined, the positional relationship between the object in the captured image and the object in the physical space of the camera viewpoint corresponds. That is, the first reference coordinate system defined for the feature points of the object in the captured image is equivalent to the coordinate system defined for the feature points of the object in the physical space of the camera viewpoint. Therefore, by displaying the display area according to the second reference coordinate system determined based on the first reference coordinate system, the content can be displayed at a desired position in the physical space.

However, when the field of view of the camera 111 moves, the positional relationship between the object in the captured image at the time of step S103 and the object in the physical space at the current camera viewpoint no longer corresponds. That is, the first reference coordinate system determined in step S103 is no longer equivalent to the coordinate system defined for the feature points of the object in the physical space. Therefore, since the positional relationship between the object in the physical space and the display area based on the second reference coordinate system is shifted, correction is required.

For example, in order to make the image follow the movement of the head, the correction is performed, for example, multiple times per second. Specifically, it is about 30 times/second or 60 times/second, and the higher the number, the better from the viewpoint of followability. Therefore, the predetermined time of S104 (and S110) is determined in consideration of these.

In order to correct the reference coordinate system, the image display device 100 acquires sensor information (S105). The controller 107 of the image display device 100 sends a sensor information acquisition command to the sensor section 102 , the sensor section 102 acquires the sensor information according to the command, and the sensor section 102 transmits the sensed sensor information to the controller 107 .

In an example of the image display device 100 shown in FIG. 4, the image display device 100 includes a gyro sensor as the sensor 112, and the gyro sensor measures the angular velocity of the image display device 100. FIG. The gyro sensor transmits measured angular velocity data to the controller 107 .

In the example of the image display device 100 shown in FIG. 4, the image display device 100 includes a gyro sensor as the sensor 112, but this embodiment is not limited to this. For example, the image display device 100 may include an acceleration sensor as the sensor 112 , the acceleration sensor may measure the acceleration of the image display device 100 , and the acceleration sensor may transmit the measured acceleration data to the controller 107 .

Next, the image display device 100 recognizes the amount of relative deviation and determines the correction coordinate system (S106). The controller 107 of the image display device 100 sends the sensor information and command received from the sensor unit 102 to the coordinate processing unit 104, and the coordinate processing unit 104 recognizes the relative deviation amount 135 and determines the correction coordinate system, and the coordinate system information to the controller 107 .

In one example, the sensor information of S105 is time-series data after S103 (for example, time-series data on how the angular velocity changed after S103). Alternatively, the data at the timing of S105 may be used assuming that the value of the sensor information is substantially constant for a short period of time. Any data can be used to calculate the relative displacement amount 135 after S103.

In other words, based on the sensor information received from the controller 107, the coordinate processing unit 104 determines the positional relationship between the position of the feature point 132 in the physical space and the video display device 100 in the reference coordinate system by moving the video display device 100. Calculate how far it is off from time.

From another aspect of this embodiment, the coordinate processing unit 104 changes the positional relationship between the position of the feature point 132 in the physical space and the video display device 100 based on the sensor information received from the controller 107 by moving the video display device 100. , how much the coordinate system information is shifted from the time when the coordinate system information was calculated last time is calculated.

As shown in FIG. 5B, by accumulating the amount of deviation from when the coordinate system information was calculated last time from when the reference coordinate system was determined, the amount of relative deviation 135 from when the reference coordinate system was determined is calculated. can be calculated.

In the example of the image display device 100 shown in FIG. 4, the coordinate processing unit 104 receives angular velocity data from the controller. Assume now that the angular velocity is substantially constant, and use the angular velocity data at the timing of S105. Let ΔT be the time that has passed since the previous calculation of the coordinate system information, and ω be the angular velocity. By accumulating the amount of angular deviation from when the previous coordinate system information was calculated from the time the reference coordinate system was determined, it is possible to calculate the amount of angular deviation from the time the first reference coordinate system was determined. be.

The coordinate processing unit 104 determines a correction coordinate system based on the recognized relative displacement amount 135 . Specifically, the coordinate processing unit 104 has information on the relationship between the relative displacement and the captured image of the imaging unit 101 in advance. In other words, information is provided in advance as to what distance or angle in what direction the relative displacement in the physical space corresponds to in the captured image of the imaging unit 101 . Such information can be determined based on the specifications of the imaging unit 101 and the specifications of the sensor unit 102 . Based on the information and the relative deviation amount, the coordinate processing unit 104 shifts or rotates the first reference coordinate system by an amount corresponding to the relative deviation amount, and determines the first correction coordinate system. As a result, even when the image display device 100 moves, the first corrected coordinate system becomes a coordinate system that does not substantially move in the physical space.

In the example shown in FIG. 5C, the x'-axis 136 and y'-axis 137 are shifted by the relative displacement amount 135 with respect to the first reference coordinate system determined by the x-axis 133 and the y-axis 134. Determine as the first correction coordinate system.

The coordinate processing unit 104 uses the coordinate conversion information to convert the first corrected coordinate system set in the captured image into the second corrected coordinate system in the video displayed by the video display unit 106 . The coordinate processing unit 104 transmits second corrected coordinate system information in the image displayed by the image display unit 106 to the controller 107 .

In the above explanation, the amount of deviation from the time when the coordinate system information was calculated last time is integrated from the time when the reference coordinate system is determined to calculate the amount of relative deviation from when the reference coordinate system was determined. Examples are not limited to this. The coordinate processing unit 104 regards the amount of deviation from when the previous coordinate system information was calculated as the amount of relative deviation, and calculates the corrected coordinate system from the reference coordinate system or the corrected coordinate system when the previous coordinate system information was calculated and the amount of relative deviation. It is also possible to configure to determine That is, the difference from the determination of the corrected coordinate system in the previous S106 is corrected in the next S106. This eliminates the need to integrate the amount of deviation, making it possible to reduce the processing.

In the above description, the coordinate processing unit 104 sets the first corrected coordinate system in the captured image, and then converts it to the second corrected coordinate system in the video displayed by the video display unit 106 according to the coordinate conversion information. However, this embodiment is not limited to this. The coordinate processing unit 104 uses the coordinate transformation information to calculate the direction and distance in the image displayed by the image display unit 106 that the relative deviation amount corresponds to, and calculates the direction and distance in the image displayed by the image display unit 106 A second correction coordinate system may be set. Thereby, it is possible to reduce the processing in the coordinate processing unit 104 .

As described above, the first reference coordinate system is set based on the feature points in the captured image captured by the camera 111, and the second reference coordinate system corresponding to this is set in the video displayed by the video display unit 106. set to Based on the signal from the sensor 112, the amount of relative deviation caused by the position and angular displacement of the camera 111 is calculated. As shown in FIG. 5, the relative displacement amount 135 corresponds to displacement of an object (typically a reference point) in the captured image. The second reference coordinate system is corrected based on this relative deviation amount. The position of the content displayed by the video display unit 106 is determined based on the corrected second reference coordinate system. By using the corrected coordinate system of , it is possible to suppress changes in the relative positional relationship between the object and the content in the captured image recognized by the user 700 .

Recognition of the amount of relative deviation and determination of the correction coordinate system do not require spatial recognition of the entire physical space, so high-speed recognition of the amount of relative deviation and determination of the correction coordinate system are possible. Note that it is possible to suppress changes in the relative positional relationship between the object and the content in the captured image by repeating the decision of the reference coordinate system (S103) itself at a high frequency. There is a problem that the processing load of

Next, the video display device 100 determines display content (S107). The controller 107 of the image display device 100 sends the coordinate system information and commands received from the coordinate processing unit 104 to the image display processing unit 105, and the image display processing unit 105 determines display content to be displayed by the image display device 100. Send the display content to the controller 107 . Determination of display content will be described later with reference to FIG.

Based on the coordinate system information received from the controller 107, the video display processing unit 105 determines the position of the content to be displayed with reference to the second corrected coordinate system. Even if the image display device 100 moves, the second correction coordinate system is a coordinate system that does not substantially move in the real space, so image display that is substantially unaffected by the movement of the image display device 100 is possible.

Next, the video display device 100 displays the content (S108). The controller 107 of the image display device 100 sends the display content and command received from the image display processing unit 105 to the image display unit 106, and the image display unit 106 displays the display content.

Next, the video display device 100 determines whether there is an end command (S109). As the end command, for example, information input from an input unit (not shown) of the image display device 100, or information obtained by recognizing the information can be used. For example, the video display device 100 has a microphone as an input unit, the microphone receives a voice command from the user 700, the video display device 100 recognizes the voice command, and when a predetermined command is recognized, the end command is issued. It is possible to configure the video display device 100 that determines that there is. If there is an end command, the video display device 100 ends the flow.

If there is no end command, the video display device 100 repeats the judgment as to whether the predetermined time has passed (S110). In other words, it waits until the predetermined time elapses. After a predetermined period of time has passed, the image display device 100 repeats the flow from acquisition of sensor information (S105). As a result, determination of the correction coordinate system by recognizing the relative deviation amount based on the sensor information can be repeated at predetermined time intervals, and correction of the coordinate system can be continued even when the image display device 100 continues to move. Thus, it is possible to continuously display an image that is substantially unaffected by movement of the image display device 100 .

FIG. 7 is a diagram showing an example of another flowchart relating to content display according to the present embodiment. Differences from FIG. 6 will be described. The image display device 100 repeats the determination of whether the predetermined time has passed until the predetermined time has passed in S110, and then determines whether the predetermined number of times has passed (S111). Specifically, for example, the controller 107 has a loop counter, increases the loop counter by a predetermined value each time a determination is made, and determines whether or not the loop counter exceeds the predetermined value.

From another aspect of the present embodiment, the image display device 100 determines whether or not a predetermined time has passed since the previous determination of the reference coordinate system, and if the predetermined time has passed, a predetermined number of times has passed. regarded as a thing. If the video display apparatus 100 determines that the predetermined number of times has elapsed, the process repeats from acquisition of the camera image (S101), and if it does not determine that the predetermined number of times has elapsed, the process repeats from the acquisition of the sensor information (S105).

As a result, the image display apparatus 100 can determine the reference coordinate system using the camera image every predetermined number of times or every predetermined period of time. etc. may occur. It is possible to improve the positional accuracy of video superimposition even with respect to such signal drift. The reference coordinate system can be updated, for example, at a frequency of about once per second.

FIG. 8 is a diagram showing an example of feature points 132 according to this embodiment. 8 shows a distribution board 131A as an example of the object 131. As shown in FIG. As the feature points 132, vertices 151A to 151D of the object 131 can be used, for example.

From another aspect of the present embodiment, sides 152A to 152D of the object 131 can be used as feature points 132, for example. From another point of view of this embodiment, the center 153 of the object 131 or the center of gravity of the object 131 can be used as the feature point 132 . From another aspect of this embodiment, it is possible to use a predetermined position of an object 154 inside the object 131 as the feature point 132 . For example, it is possible to use the vertex 154A of the object 154, the side 154B of the object 154, and the like.

From another point of view of this embodiment, it is possible to use a predetermined position 155A of the character string 155 written on the object 131 as the feature point 132 . As the predetermined position 155A, for example, the topmost, bottommost, rightmost, and leftmost positions among the characters of the predetermined number of characters in the character string 155 can be used. It is possible.

In another aspect of this embodiment, the feature points 132 can be predetermined symbols 156 . The predetermined symbol may be a predetermined symbol or an unknown symbol. The symbol 156 may be a one-dimensional code, a two-dimensional code, or the like. The feature point 132 may be a component of the object 131 that the object 131 has in advance, or may be an object attached to the object 131 .

Feature points can be defined in advance as described above, and the feature point position recognition unit 103 recognizes the color (e.g., hue, brightness, and saturation) and shape of the feature point from the captured image captured by the camera 111. can be extracted by a known method based on the

FIG. 9 is a diagram showing an example of a display content determination method by the video display processing unit 105 according to the present embodiment. For the sake of explanation, FIG. 9 shows both the real space and the virtual space as viewed by the user 700 . An object 131 is an object located in the physical space. The coordinate processing unit 104 determines the x'-axis 136 and the y'-axis 137 in the captured image as the first corrected coordinate system, and converts them into the second corrected coordinate system in the video displayed by the video display unit 106.

Now, as a specific example for helping understanding, the object 131 is an object in the real space, such as a distribution board. In this example, the field of view of the camera 111 may be large enough to recognize the object 131 . In this example, a case will be described where it is desired to display content 162, for example, an operation manual for a distribution board, on the right side of object 131. FIG.

Based on the coordinate system information received from the controller 107, the video display processing unit 105 determines the position of the content to be displayed with reference to the second corrected coordinate system. Specifically, the video display device 100 determines the position of the display content, for example, as follows.

The video display processing unit 105 stores contents in the virtual space in advance in a virtual space storage unit (not shown). For example, the RAM 115 can be used as the virtual space storage unit. A plurality of content images are stored in the virtual space 161 stored in the virtual space storage unit while being positioned at predetermined positions in the virtual space 161 .

In other words, as for the virtual space, it has a map indicating which contents are to be displayed at which locations with respect to the object 131 . Specifically, in this example, the relative positional relationship between the object 131 and the virtual space 161 is determined in advance, and the content 162 is determined to be positioned on the right side of the object 131 . The position of the content can be indicated by coordinates according to the second coordinate system, for example. The size of the virtual space 161 is arbitrary and the size is not limited. It should be noted that the physical space does not have a map, and it is sufficient if the first reference coordinate system can be generated based on the camera image.

In the example of FIG. 9, three contents 162A to 162C are stored in the virtual space 161 with their positions shifted in the horizontal direction. The storage format of the virtual space 161 is, for example, an image file such as a bitmap image. In a general example, since the angle of view that can be displayed by the image display unit 106 is small, the area that can be displayed by the image display device 100 is only the content located in the display area 163 in the virtual space 161. The position of the area 163 is determined by the video display processing unit 105 . In the example of FIG. 9, the display area 163 overlaps part of the content 162A, and the video display device 100 can display part of the content 162A within the display area 163. In the example of FIG.

The position of the display area 163 is determined by the coordinates of the second coordinate system in conjunction with the movement of the head, for example. In this way, the display area is cut out from the virtual space and the image is displayed in accordance with the movement of the camera. As for the method of viewing the content arranged in the line-of-sight direction by changing the direction of the head, for example, the known technology described in Patent Literature 1 or the like may be adopted.

As already explained, in the present embodiment, a method of correcting the second reference coordinate system itself was shown, but the same is true even if the coordinates of the display area 163 and the coordinates of the content in the second reference coordinate system are corrected. effective. In this case, the coordinates of the content or the like may be corrected so that the amount of shift is the same as in the case of correcting the second reference coordinate system, but the direction is reversed.

After determining the x′-axis 136 and the y′-axis 137 in the captured image as the first correction coordinate system, the coordinate processing unit 104 determines the positions and positions of the x′-axis 136 and the y′-axis 137 in the virtual space 161 . It calculates whether it corresponds to the direction, and transmits the coordinate system information to the controller 107 as the second correction coordinate system in the virtual space 161 . That is, the coordinate system information represents, for example, what position and what direction at least one of the virtual space 161 and the display area 163 corresponds to in the second corrected coordinate system.

The image display processing unit 105 receives coordinate system information regarding the second corrected coordinate system in the virtual space 161 from the controller 107, and sets the display area 163 at a predetermined position in the second corrected coordinate system.

The video display processing unit 105 transmits information in the display area 163 to the controller 107 as display content. Since the second corrected coordinate system is a coordinate system that does not substantially move in the physical space, the content displayed by the video display device 100 is substantially unaffected by the movement of the video display device 100 . That is, it is possible to display an image that the user 700 perceives as if the content were pasted on the real space. In addition, since the contents are stored in advance in the virtual space storage unit, it becomes unnecessary to draw a part or the whole of the virtual space each time the video display processing unit 105 determines the display content. It is possible to speed up, that is, to increase the speed.

In a transmissive display in which content is superimposed on the real space and visually recognized, display content can be simultaneously displayed when a user visually recognizes an object as described above. In the above description, both the reference coordinate system and the correction coordinate system are defined on the object in the photographed image, so basically the two-dimensional coordinate system may be used. The same applies to the coordinate system of the virtual space.

FIG. 10 is a diagram showing another example of the display content determination method by the video display processing unit 105 according to the present embodiment. For the sake of explanation, FIG. 10 shows both the real space and the virtual space seen by the user 700 . The image display processing unit 105 is provided in advance with information regarding the coordinates in the corrected coordinate system of the content 171 displayed by the image display device. For example, the video display processing unit 105 has information such as to display a content called an operation manual at a position 5 degrees from the right end of the distribution board. This information can be expressed as coordinates in a second corrected coordinate system. When recognizing that at least part of the content 171 overlaps the display area 163 , the video display processing unit 105 draws the content 171 as display content and transmits the display content to the controller 107 .

Since the second corrected coordinate system is a coordinate system that does not substantially move in the physical space, the image display device 100 is not substantially affected by the movement of the image display device 100, that is, the content can be used as if it were attached to the physical space. A video display that is perceived by the person 700 is possible. For example, when the corrected coordinate system is attached to the distribution board, by specifying the coordinates in the corrected coordinate system, the content can be displayed as if it were attached to the distribution board. In addition, since it is not necessary to store the information of the entire virtual space in the virtual space storage unit, it is possible to reduce the usage amount of the storage unit and reduce the cost.

FIG. 11 is a diagram showing an example of positions of display content displayed by the video display device 100 according to the present embodiment. The display position of the display content can be determined in advance by determining the positional relationship with the object in the first correction coordinate system by the following method and converting it into the coordinates of the second correction coordinate system using the coordinate conversion information. If the position of the content 162 in the virtual space 161 and the position of the display area 163 are determined in the second correction coordinate system, the overlapping portion is displayed.

An eye point 180 indicates the eye position of the user 700 who uses the video display device 100 . Also, a point on the object 131 visually recognized by the user 700 when the user 700 faces the front is defined as a center point 181 , and an end of the object 131 is defined as an edge 182 .

The video display device 100 displays, for example, the display content 184 so as not to overlap the object 131 . Specifically, the display content 184 is displayed such that the edge 183 of the display content 184 is on the opposite side of the center point 181 with respect to the edge 182 of the object. As a result, it is possible to prevent the display content 184 from making it difficult for the user 700 to recognize the object 131, and the visibility of the object 131 can be improved. In another aspect of this embodiment, center point 181 may be the center of object 131 .

From another aspect of the present embodiment, the display content is displayed so that the angle θ between the line segment connecting the eye point 180 and the center point 181 and the line segment connecting the eye point 180 and the edge 183 of the display content is greater than or equal to a predetermined angle. View content. For example, the video display device 100 displays the display content so that the angle θ is 2.5 degrees or more. As a result, when the user 700 is looking at the front, the image display device 100 can display an image while avoiding the range of the discrimination visual field, thereby improving the visibility of the object 131 by the user 700. can be done.

From another aspect of this embodiment, the video display device 100 displays the display content so that the angle θ is 15 degrees or more. As a result, the image display device 100 can display an image while avoiding the effective field of view, so that the visibility of the object 131 by the user 700 can be further improved.

From another aspect of this embodiment, the video display device 100 displays the display content so that the angle θ is 30 degrees or more. As a result, the image display device 100 can display an image while avoiding the stable fixation field, so that the visibility of the object 131 by the user 700 can be further improved.

From another aspect of this embodiment, the video display device 100 displays the display content so that the angle θ is 50 degrees or more. As a result, the image display device 100 can display an image while avoiding the induced visual field, so that the visibility of the object 131 by the user 700 can be further improved.

According to this embodiment, the reference coordinate system is determined based on the captured image of the imaging unit 101 and the correction coordinate system is determined based on the sensing information from the sensor unit 102, and the display content is displayed based on the correction coordinate system. is determined, it is possible to provide the image display device 100 capable of displaying images with high positional accuracy. In addition, since it is possible to recognize the position of the feature point and the relative deviation amount at high speed, determine the reference coordinate system and the correction coordinate system, and determine the display content, the superimposed image with little delay with respect to the movement of the image display device 100 can be provided.

In the second embodiment, the feature point for determining the reference coordinate system can be changed with time. In addition, in Example 2, differences from the above-described examples will be mainly described, and the same components as those of the above-described examples will be given the same reference numerals, and their description will be omitted.

FIG. 12 is a diagram showing an example of feature points and a reference coordinate system according to this embodiment.

FIG. 13 is a diagram showing an example of a flowchart relating to content display according to the present embodiment. Hereinafter, the parts different from FIG. 7 will be mainly described.

In the flow shown in FIG. 13, when it is determined in S111 that the predetermined number of times has passed, recognition of the position of the feature point (S202) and determination of the reference coordinate system (S203) are repeated. Time T1, time T2, and time T3 are times at which the position is recognized (S202) and the reference coordinate system is determined (S203). Here, the time T2 is the time after the time T1, and the time T3 is the time after the time T2.

There is naturally a time difference between the time when the position of the feature point is recognized (S202) and the time when the reference coordinate system is determined (S203). The time difference between determination (S203) is not particularly mentioned, and a typical time at which both steps are executed is referred to as time T1 or the like.

As shown in FIG. 12A, in S202 at time T1, the feature point position recognition unit 103 recognizes the position of the feature point 200. As shown in FIG. In S203 at time T1, the coordinate processing unit 104 determines the x-axis 202A and the y-axis 203A of the reference coordinate system based on the position of the feature point 200. FIG.

In S202 at time T2, the feature point position recognition unit 103 recognizes the positions of the feature points 200 and 201, and transmits the recognition result of the position of the feature points 200 and 201 different from the feature points 200 to the controller 107. . Although two feature points are recognized in this example, three or more feature points may be recognized.

As shown in FIG. 12B, in S203 at time T2, the controller transmits the recognition results of the positions of the feature points 200 and 201 to the coordinate processing unit 104. As shown in FIG. The coordinate processing unit 104 detects a positional difference 204 between the feature point 200 and the feature point 201, and the x-axis 202A and y-axis 203A of the reference coordinate system determined at time T1, or determined by the coordinate processing unit 104 immediately before time T2. x'-axis and y'-axis of the corrected coordinate system and the x-axis 202B and y-axis 203B of the reference coordinate system with the feature point 201 as a reference are substantially aligned with each other. Determine axis 202B and y-axis 203B.

As shown in FIG. 12(c), the feature point position recognition unit 103 recognizes the position of the feature point 201 at S202 at time T3. In S203 at time T3, the coordinate processing unit 104 determines the x-axis 202B and the y-axis 203B of the reference coordinate system based on the position of the feature point 201. FIG.

According to this embodiment, the image display device 100 can change the feature points for determining the reference coordinate system. As a result, as the image display device 100 moves, even if the feature point in the image captured by the imaging unit 101 goes out of the range of the image, another feature point captured in the image captured by the imaging unit 101 can be detected. can be used to determine the reference coordinate system. Therefore, it is possible to provide the image display device 100 capable of displaying images with high positional accuracy even when the image display device 100 moves greatly.

In the third embodiment, the configuration is such that the user 700 can indicate a feature point with a command. In addition, in Example 3, differences from the above-described examples will be mainly described, and the same components as those of the above-described examples will be given the same reference numerals, and descriptions thereof will be omitted.

FIG. 14 is a diagram showing an example of functional blocks of the video display device 100 according to this embodiment. The video display device 100 includes an input section 301 and a command recognition section 302 .

FIG. 15 is a diagram showing an example of the video display device 100 according to this embodiment. The video display device 100 includes a microphone 310 as an input section 301 .

FIG. 16 is a diagram showing an example of a flowchart leading to determination of the reference coordinate system in this embodiment.

First, the image display device 100 displays a marker (S301). The controller 107 transmits commands to the video display processing unit 105 . The video display processing unit 105 draws a marker on the display content and transmits the display content to the controller 107 according to the command.

FIG. 17 is a diagram showing an example of how markers are superimposed and displayed in the physical space according to the present embodiment. A marker 321 is displayed in part of the image area 320 displayed by the image display device 100 . FIG. 16 shows an example of using a "+" symbol as a marker.

Next, the video display device 100 determines whether or not there is a command designating a feature point (S302). The input unit 301 transmits information input to the input unit to the controller 107 . The controller transmits information input from the input unit to the command recognition unit 302 . A command recognition unit 302 detects whether or not a predetermined command exists in the information received from the controller. Command recognition unit 302 transmits the detection result to controller 107 .

In an example of a video display device that includes a microphone 310 as the input unit 301, the command recognition unit 302 includes, for example, a predetermined character string uttered by the user 700, such as "feature point set," in sound information sensed by the microphone. detect if it is If a predetermined character string is included, the command recognition unit 302 transmits the recognition result to the controller 107 as a command specifying a feature point.

The controller 107 repeats until it receives a recognition result indicating that there is a command specifying a feature point from the command recognition unit 302 .

Next, the video display device 100 hides the marker (S303). The controller 107 transmits commands to the video display processing unit 105 . In accordance with the command, the video display processing unit 105 changes the operation so as not to draw the marker on the display content, and transmits the display content to the controller 107 .

Next, the video display device 100 acquires a camera image (S101).

Next, the image display device 100 recognizes the position of the feature point (S304). The controller 107 transmits the predetermined position 322 of the marker 321 to the feature point position recognition unit 103, and the feature point position recognition unit 103 uses the marker position 322 in the camera image as a feature point. Then, the position of the feature point within the camera image is recognized.

Although the position 322 of the marker is used as a feature point in the above description, the present invention is not limited to this. For example, the vicinity of the marker can be set as the feature point. For example, it is possible to use an object existing within a predetermined pixel range around the position of the marker in the camera image as a feature point. This makes it possible to set feature points even when there is no object at the position of the marker.

In another aspect of the present invention, for example, the location indicated by the marker or the vicinity thereof can be used as the feature point. For example, it is possible to display an arrow as a marker and use the tip of the arrow or its vicinity as a feature point. By this. It is possible to prevent the marker from overlapping with the object, and it is possible to improve the visibility.

The above flow is the feature point position recognition flow of the video display apparatus 100 according to the present embodiment, and after the feature point position recognition (S304), the steps after S104 in FIG. 7 and the like are executed.

According to this embodiment, it is possible for the user 700 to indicate a feature point with a command, and it is possible to determine the reference coordinate system according to the intention of the user 700 . In addition, even when it is difficult for the image display device 100 to determine the feature points, it is possible to provide the image display device 100 capable of stably displaying images with high positional accuracy by designating the feature points according to the user's intention. is.

Example 4 is an example that is utilized in a remote work support system 450 in factories and sites where workers perform assembly, manufacturing, inspection, maintenance, inspection, and the like.

FIG. 18 is a configuration diagram showing an example of a remote work support system 450 utilizing the video display device 100 according to this embodiment.

A worker 701 carries the image display device 100 of the present embodiment, and performs product assembly, inspection, and the like on a factory line. The image display device 100 projects a virtual image onto the space as described above, and displays contents for work support, such as work procedure manuals, drawings, and checklists. In the embodiment of the present invention, by determining the spatial position coordinates of the range display on the visual field that does not interfere with the manual work of the worker 701, it is possible to always display the image, and it is equipped with a technology that greatly improves usability. can be For example, horizontal ±10° to ±75° and vertical -60° to -90° from the center of the line of sight can be set not to be displayed automatically or by the worker 701 side or the manager 420 side. It is possible to safely secure and confirm the field of view at hand and feet without displaying content, and work can be carried out safely and securely even during inspection work and assembly work that requires moving around.

The image display device 100 is equipped with a communication means (not shown), and is remotely located via a network 401 such as Wi-Fi (trademark), LTE (trademark), 5G, 6G, wired, optical, local, public wireless, or the like. It is connected to the remote management device 410 of the administrator 420 and shares information. A worker 701 owns a video display device 100 such as a tablet, smartphone, HMD, or PC (Personal Computer), which is an edge terminal, and exchanges data through the network 401 . Via the network 401 , the image of the camera 402 around the worker, the data of the sensor 403 , the sensing data such as the state of the device 404 , etc. are transmitted to the edge server 407 and the administrator 420 . The edge server 407 and administrator 420 carry out situation recognition, judgment, and prediction (particularly prediction of danger, abnormality, and abnormal behavior) based on various information. Also, when an abnormal state is recognized in this way, the edge server 407 and the administrator 420 give work instructions and alerts to the worker 701 through display images, sounds, and instruction signals on the image display device 100 and the like.

In addition, when the worker 701 asks questions about drawings and work processes, and incorporates a work management system (not shown) that automatically discriminates check errors in the work process, the worker 701 or the manager 420 can monitor the site in real time. Display content corresponding to the situation can be displayed on the video display device 100 such as an edge terminal. Furthermore, contents supporting work instructions such as work instructions, character input, and handwritten input can be displayed by the administrator 420, and questions, questions, and worries of the worker 701 can be resolved by the displayed contents. At this time, the display content is displayed at the above-described absolute coordinate position of the line of sight of the worker 701, or at an absolute coordinate or relative coordinate arrangement that does not interfere with the line of sight of the worker 701. Since it is displayed in a position that does not get in the way, a more comfortable remote work support system 450 can be provided.

By superimposing the image of the virtual space on the real space in this way, it is possible to provide a user-friendly image display device 100 such as a head-mounted display that presents information to the user and the remote work support system 450 . In factories, etc., there are cases where work is performed while viewing content such as the work process, but when it is difficult to place paper media such as displays, drawings, and work process charts near the work target, or when it is difficult to place both hands in a high place There are tasks that require In such a case, if the video display device 100 such as a head-mounted display and the remote work support system 450 are used, the worker 701 can perform the work while referring to the work instructions displayed on the video display device 100. By utilizing the above-described remote work support system 450, which is excellent in usability, it is possible to improve safety and security and work efficiency.

The administrator 420 may be located inside the same network 401 as the worker 701, or may be located in another location via the Internet. Also, at least part of the edge servers 407 described above may be cloud servers 408 . The remote work support system 450 may be provided in the edge server 407 or may be provided in the cloud server 408 .

FIG. 19 is an example of a schematic configuration diagram of a video transmission system incorporated in the remote work support system 450 of this embodiment. For example, a 4K, 30 fps camera is used as the camera 402 that observes the work environment of the worker 701 . USB 3.0 is used for information transmission from the camera to the PC 431 . Also, the PC 431 utilizes the Wi-Fi6 router 432 (IEEE 802.11ax, 9.6 Gbps) to transmit video information. It also has a function of bi-directionally or uni-directionally focused communication of video information, audio, sensor information, etc. to the administrator 420 located on the remote side via the Wi-Fi6 network 433 . The Wi-Fi router 434 installed on the remote management side is used to connect to the PC 435 . For example, the connection from the Wi-Fi router 434 to the PC 435 is 1000BASE-T (IEEE 802.3ab) to exchange information. As a result, video information and audio information can be smoothly communicated with a low delay of 200 ms or less.

As described above, the remote work support system 450 capable of transmitting at least two or more pieces of 4K video information and audio information at high speed and low delay is realized by the information transmission system of FIG. Spatial display can be achieved within 100ms. Therefore, the total amount of delay is 300 ms or less, and a delay time of 300 ms or less that humans perceive can be realized. As a result, a 4K video transmission system with a low delay of 300 ms or less can be constructed, and a remote work support system 450 with improved usability can be realized.

FIG. 20 is a diagram showing an example of a remote work support system 450. As shown in FIG. A worker 701 carries the image display device 100 with him. The image display device 100 has a camera (not shown). The camera captures an image in front of the worker 701 , and the captured image is transmitted to the remote control device 410 of the manager 420 via the network 401 . Remote management device 410 displays the camera image, and administrator 420 visually recognizes the displayed image 460 .

At least an object 131 is captured in the image captured by the camera. The display image 460 is displayed on, for example, a tablet capable of touch operation or handwriting input. The manager 420 recognizes the displayed image and inputs instructions to the worker 701 by touch operation or handwritten input. For example, the worker 701 inputs by indicating a predetermined position where the work should be performed with a circle or an arrow. The input instruction is transmitted to the image display device 100 via the network 401, and the image display device 100 displays the instruction.

The video captured by the camera may change dynamically due to the movement of the worker 701 or the like. In this case, the remote work support system 450 includes an indicated position recognition unit (not shown), which recognizes the object 131 appearing in the image captured by the camera and recognizes the instruction input by the user. The position and the relative position of the object 131 are recognized, and the position of the instruction to be displayed by the video display device 100 is transmitted via the network 401 as the relative position to the object 131 . The image display device 100 displays instructions based on the relative position with respect to the object 131 . The video display processing unit 105 performs the display processing of the video displayed by the video display device 100 . That is, in this embodiment, a reference coordinate system is determined based on the captured image of the imaging unit 101 and a correction coordinate system is determined based on sensing information from the sensor unit 102, and display content is determined based on the correction coordinate system. To make decisions, the video display device 100 can display instructions with a high degree of positional accuracy.

Note that the position of the instruction displayed by the image display device 100 may be superimposed on the object 131 or may be displayed so as not to overlap the object 131 . For example, as shown in FIG. 17 , the video display device 100 may display the display content 184 so as not to overlap the object 131 .

In the above description, the worker 701 inputs by indicating the predetermined position where the work should be performed with a circle or an arrow, but the present invention is not limited to this. For example, the remote management device 410 has a microphone and a voice recognition unit, the manager 420 issues instructions to the worker 701 by voice, the microphone collects the voice of the manager 420, and the collected voice is transmitted to the voice recognition unit. may recognize and use the recognized instruction as an input.

From another aspect of the present embodiment, the administrator 420 inputs the instructions by voice as described above, and also inputs the position of displaying the instructions on the tablet or the like on which the display image 460 is displayed. may be configured to display the instructions at the displayed position.

Although the administrator 420 inputs instructions by voice in the above description, the present invention is not limited to this. For example, a predetermined instruction may be obtained from a database included in the edge server 407, the cloud server 408, the remote management device 410, or the remote work support system 450, and the obtained instruction may be used as an input.

In the above embodiment, it is assumed that the position of the display area 163 is determined according to the direction of the head of the user 700 . In this case, the virtual space is fixed to the real space. However, as described in Patent Document 1, the position of the display area 163 can also be determined based on the difference between the movements of the user's 700 head and waist. In this case, when viewed from the user 700, the virtual space always moves in the direction of the waist.

According to the above embodiment, with a small amount of processing, there is little delay in following the motion of the user's head, and it is possible to superimpose the real space and the virtual space with high position accuracy, so energy consumption is low and carbon emissions are low. It can reduce the amount, prevent global warming, and contribute to the realization of a sustainable society.

The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the scope of the invention. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to those having all the configurations described. Also, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Moreover, the configuration of another embodiment may be added to the configuration of one embodiment. Further, additions, deletions, and replacements of other configurations may be made for a part of the configuration of each embodiment.

In addition, each configuration, function, processing unit, processing means, etc. described above may be realized by hardware, for example, by designing a part or all of them with an integrated circuit, and the processor realizes each function. It may be realized by software by interpreting and executing a program to execute.

Information such as programs, tables, and files that implement each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD memory cards, and DVDs. .

In addition, the control lines and information lines indicate those considered necessary for explanation, and do not necessarily indicate all the control lines and information lines necessary for mounting. In practice, it can be considered that almost all configurations are interconnected.

100: video display device, 101: imaging unit, 102: sensor unit, 103: feature point position recognition unit, 104: coordinate processing unit, 105: video display processing unit, 106: video display unit, 107: controller, 111: camera , 112: sensor, 113: display, 114: CPU, 115: RAM, 116: storage medium, 121: gyro sensor, 122: acceleration sensor, 131: object, 131A: distribution board, 132: feature point, 133: x Axes 134: y-axis 135: relative displacement amount 136: x'-axis 137: y'-axis 151: vertex 152: side 153: center 154: object 154A: vertex 154B: side 155 : character string, 155A: predetermined position, 156: symbol, 161: virtual space, 162: content, 163: display area, 171: content, 180: eye point, 181: center point, 182: edge, 183: edge, 184: display content, 200: feature point, 201: feature point, 202: x-axis, 203: y-axis, 204: position difference, 301: input unit, 302: command recognition unit, 310: microphone, 320: image area , 321: marker, 322: position, 401: network, 402: camera, 403: sensor, 404: device, 407: edge server, 408: cloud server, 410: remote management device, 420: administrator, 450: remote work Support system, 460: Display image, 700: User, 701: Worker

Claims (15)

In the image display system,
a feature point position recognition unit;
a coordinate processing unit;
and a video display processing unit,
The feature point position recognition unit recognizes the position of the feature point of the object based on the captured image captured by the imaging unit,
The coordinate processing unit determines a reference coordinate system based on the positions of the feature points, detects the movement of the image capturing unit based on sensor information indicated by the sensor unit, and determines the display position of the image by the movement of the image capturing unit. corrected based on
The image display system, wherein the image display processing unit displays the image at a corrected image display position when the image is displayed based on the reference coordinate system. The video display system according to claim 1,
The coordinate processing unit converts a first reference coordinate system in the captured image into a second reference coordinate system based on coordinate conversion information regarding a positional relationship between the captured image and a video to be displayed,
The image display system, wherein the image display processing unit displays an image based on the second reference coordinate system. The video display system according to claim 1,
The coordinate processing unit corrects the display position of the image based on the movement of the imaging unit by correcting the reference coordinate system to determine a correction coordinate system,
The video display system, wherein the video display processing unit displays the content at a predetermined position in the corrected coordinate system. The video display system according to claim 1,
The image display unit for displaying the image, the image display processing unit, the imaging unit, the sensor unit, the feature point position recognition unit, and the coordinate processing unit are included in a wearable image display device. video display system. The video display system according to claim 1,
The image display system, wherein the sensor unit includes at least one of a gyro sensor and an acceleration sensor. The video display system according to claim 1,
The image display system according to claim 1, wherein the coordinate processing section corrects the display position of the image based on the movement of the imaging section after the reference coordinate system is determined. The video display system according to claim 1,
The coordinate processing unit determines and updates the reference coordinate system again based on the positions of the feature points after a predetermined time or more has elapsed from the time when the reference coordinate system was determined based on the positions of the feature points. An image display system characterized by: The video display system according to claim 1,
The feature point position recognition unit recognizes the first feature point and the recognizing the position of a second feature point different from the first feature point;
The coordinate processing unit determines and updates a reference coordinate system based on the positions of the second feature points recognized by the feature point position recognition unit at time T2,
The feature point position recognition unit recognizes the position of the second feature point at time T3 after a predetermined period of time or more has elapsed from time T2,
The image display system, wherein the coordinate processing unit determines and updates the reference coordinate system based on the position of the second feature point recognized by the feature point position recognition unit at time T3. The video display system according to claim 1,
The feature point is an edge or corner of the object in the captured image, or a point in the captured image where at least one of hue, brightness, and saturation changes more than a predetermined amount, or a predetermined object. A video display system characterized by: The video display system according to claim 1,
an input unit;
a video display unit that displays a video superimposed on the captured image;
a command recognition unit that recognizes a command input to the input unit;
After displaying a predetermined video superimposed on the captured image, when the command recognition unit recognizes a predetermined command, the feature point position recognition unit recognizes the location of the predetermined video in the captured image, the vicinity of the location, 1. An image display system, wherein either a location indicated by said image or a vicinity of said location indicated by said image is defined as a feature point and the position of said feature point is recognized. In the video display system according to claim 3,
The image display system, wherein the predetermined position is a position away from the front of the user by a predetermined angle or more, or a position away from the feature point with respect to the front of the user. In a video display device that displays video,
an imaging unit that captures an object and acquires a captured image;
a sensor unit that detects the movement of the imaging unit;
a feature point position recognition unit;
a coordinate processing unit;
a video display processing unit;
and a video display unit that displays video,
The feature point position recognition unit recognizes the position of the feature point of the object based on the captured image,
The coordinate processing unit determines a reference coordinate system based on the position of the feature point, and generates correction information for the position of the image displayed by the image display unit based on the movement of the imaging unit,
The video display processing unit determines a display position of the video based on the reference coordinate system and the correction information,
the image display unit displays the image at the determined display position;
An image display device characterized by: A video display method for displaying desired content,
a first step of recognizing the positions of the feature points of the object in the image taken by the camera;
a second step of determining a reference coordinate system based on said feature points;
a third step of detecting movement of the camera;
a fourth step of correcting the display position of the content based on the movement of the camera when displaying the content based on the reference coordinate system;
A video display method that executes In the image display method according to claim 13,
The correction includes at least one of a process of correcting the reference coordinate system to a correction coordinate system and a process of correcting the coordinates of the content.
Image display method. In the image display method according to claim 13,
The camera and a sensor for detecting movement of the camera are provided in a video display device worn by a user,
the content is simultaneously displayed when the user views the object;
Image display method.

Images