JP2012160898A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform image processing on a camera image in which an object in an actual external world is imaged by a camera 21 attached to the head of an observer together with an HMD 12 so that the object is appropriately displayed by a monitoring device 130 when the monitoring device 130 displays the camera image to a person different from the observer.SOLUTION: An image processing apparatus performs image inclination correction by detecting an inclination angle of the head of the observer with respect to a gravity direction, and correcting an inclination of the camera image with respect to the gravity direction in the reverse direction with respect to the direction of the detected inclination angle, and at substantially the same angle. The image processing apparatus displays the camera image after being subjected to the image inclination correction as a monitor image on a screen 150 of the monitoring device 130.

Description

  According to the present invention, a monitor device displays a camera image in which an object in the real outside world is captured by a camera mounted on the head of an observer together with a head-mounted display (hereinafter abbreviated as “HMD”). In particular, the present invention relates to a technique for performing image processing on the camera image so that the object is appropriately displayed on a monitor device.

  An image display device having an HMD that is mounted on the observer's head to project image light representing an image to be displayed on the retina of the observer's eye is already known. According to this image display device, an observer can directly observe an image without using a screen on which the image is projected.

  When the HMDs are classified according to the image light forming method, the image light is emitted using light emitted from a light source using a spatial modulation type, that is, a spatial modulation element (for example, liquid crystal or organic EL) that operates according to an image signal. And a scanning type, that is, a method of forming the image light by scanning light emitted from a light source and having intensity corresponding to the image signal.

  Further, when the HMD is classified according to the display image observation method, a see-through type, that is, a method in which an observer can superimpose a display image by the HMD and observe the actual outside world, and a sealed type, that is, The system is classified into a method in which the incident of external light representing the actual outside world is completely or partially blocked (for example, only for one eye) and the observer can mainly observe only the display image by the HMD. Is done.

  When the HMD is a see-through type, the observer can observe the real outside world located in front of the HMD as a real image together with a display image by the HMD. However, even if the HMD is a hermetically sealed type, the HMD is captured so that the real external environment is imaged by a camera mounted on the observer's head and the camera image is displayed to the observer as a second display image. It is possible to configure. According to this configuration, even with a sealed HMD, an observer can observe the actual outside world located in front of the virtual HMD as a virtual image together with a display image by the HMD.

  Patent Document 1 discloses a see-through type HMD that displays an image corresponding to an image signal as a virtual image but does not use a camera that captures the actual outside world. In the HMD, the tilt angle of the head of the observer wearing the HMD is detected, and based on the detected tilt angle, the tilt of the image is opposite to the direction of the detected tilt angle. In addition, image inclination correction for correcting at substantially the same angle is performed on the display image by the HMD.

  Patent Document 2 discloses an HMD that displays to a viewer a finder image of a camera that captures a subject in the actual outside world. In this HMD, a finder image (a kind of camera image) is a display image by the HMD. This camera is not mounted on the observer's head so that the observer can take a picture with his / her hand. Therefore, the camera may move differently from the HMD. Nevertheless, the image tilt correction is performed on the display image in the HMD so that the orientation of the subject in the display image by the HMD does not change.

JP 2005-323001 A JP 2007-333929 A

  The present inventor conducted research on an image display device having an HMD, and as a result, a camera image captured by a camera mounted on the observer's head together with the HMD can be obtained in a remote place away from the observer's position. We proposed a remote support system that the monitor device can display to a different person from the observer.

  Specifically, in this remote support system, the camera mounted on the observer's head is a camera image of an object (subject) in the real external environment that the observer observes together with the display image by the HMD. As an image. In this remote support system, an image display device having an HMD and a monitor device that displays the camera image as a monitor image to a person other than the observer can communicate with each other wirelessly or by wire. It has become. According to this remote support system, it is possible for a person other than the observer to monitor the state of view of the observer and / or the state of the object in a remote place.

  As a result of further research on this remote support system, the present inventor has found that this remote support system has room for improvement. This will be described below, but will be briefly described here and will be described in detail later with reference to the drawings.

  In this remote support system, when the observer tilts his / her head, the image of the object in the camera image is tilted, and along with this tilt, the image of the object in the monitor image on the screen of the monitor device is also tilted. As a result, for a person who wants to provide some instructions or advice to the observer while watching the monitor image (hereinafter referred to as “supporter”), the monitor is tilted every time the observer tilts his / her head. In the image, the orientation of the object image changes, and as a result, the object image shakes. This can be a factor that reduces the efficiency when the supporter performs the support work by analyzing the monitor image, and can also cause a visual discomfort to the supporter.

  Furthermore, in this remote support system, when the observer tilts his / her head, the image of the object has a different orientation from the actual orientation of the object in the real external environment (absolute space). Is displayed. In contrast, when an observer tilts his / her head, the image of the object tilts relative to the observer, but the cause is that the observer has tilted himself / herself. Therefore, the observer tends to recognize that the orientation of the object does not change even when the head is tilted.

  If it does so, the shift | offset | difference of recognition will generate | occur | produce between an observer and supporter regarding the direction of a target object. Despite the existence of such a shift, if the supporter sends a message using an expression about the direction to the observer while watching the monitor image, the observer misunderstands the meaning of the expression. there is a possibility.

  Therefore, in this remote support system, even if the observer tilts his / her head, the orientation of the image of the object in the monitor image is not changed as much as possible, thereby improving the work efficiency and work comfort of the support person. Desirable to make.

  Based on such knowledge, the present invention provides a camera image obtained by capturing an image of an object in the real outside world with a camera mounted on the observer's head together with the HMD. On the other hand, it is an object of the present invention to provide a technique for performing image processing on the camera image so that the object is appropriately displayed on a monitor device when displayed.

  The following aspects are obtained by the present invention. Each aspect is divided into sections, each section is numbered, and is described in the form of quoting the section numbers of other sections as necessary. In this way, each section is referred to the section number of the other section. By describing in the format, the technical features described in each section can be appropriately made independent according to the properties.

(1) A camera image captured by a camera mounted on an observer's head together with a head-mounted display (HMD), and in an actual external environment observed by the observer together with the display image by the HMD An image processing apparatus that performs image processing on an image of an object,
The image processing device is provided in at least the image display device of the image display device and the monitor device having the HMD,
The HMD projects image light representing an image according to an image signal onto an observer's eye, thereby displaying the image as a display image in a first image display area,
The monitor device displays the camera image as a monitor image in a second image display area based on camera image data representing the camera image received wirelessly or by wire from the image display device;
The image processing apparatus
An inclination angle detector that detects an inclination angle of the head of the observer with respect to the direction of gravity and outputs inclination angle data representing the detected inclination angle;
Based on the output tilt angle data, image tilt correction for correcting the tilt of the image with respect to the gravitational direction in a direction opposite to the direction of the detected tilt angle and substantially the same angle is performed. An image tilt correction unit that performs image data but does not perform display image data representing the display image,
The monitor device displays the monitor image in the second image display area based on corrected camera image data which is camera image data on which the image tilt correction has been performed.

(2) Furthermore,
A transmission unit provided in the image display device, for transmitting the camera image data and the tilt angle data to the monitor device;
The said image inclination correction | amendment part is provided in the said monitor apparatus, Based on the inclination angle data received from the said transmission part, the said image inclination correction | amendment is performed with respect to the camera image data received from the said transmission part. Image processing apparatus.

(3) The image tilt correction unit is provided in the image display device, performs the image tilt correction on the camera image data based on the tilt angle data,
The image processing apparatus further includes:
A transmission unit that is provided in the image display device and transmits the corrected camera image data to the monitor device;
The image processing apparatus according to (1), wherein the monitor device displays the monitor image in the second image display area based on the corrected camera image data received from the transmission unit.

(4) Furthermore,
A check image display unit that is provided in the image display device and displays the corrected camera image as a check image for an observer in a sub-area arranged at a predetermined position in the first image display area ( The image processing apparatus according to any one of items 1) to (3).

(5) Furthermore,
The image display device is a see-through type in which external light representing the object is projected onto the first image display area together with display light representing the display image, or the object is imaged by the camera. Regardless of whether the display light representing both the captured image and the display image is projected onto the first image display area, the observer observes the entire image in the first image display area. Among the images including the display image and the image of the object, a frame image for visually distinguishing a matched partial image that matches the corrected camera image from a mismatched partial image is the first image. The image processing device according to any one of (1) to (3), including a frame image display unit for displaying in the image display area.

(6) The image tilt correction unit rotates the camera image before the image tilt correction around the rotation center point where the position is specified, in the direction opposite to the tilt angle and at substantially the same angle. The corrected camera image is acquired by performing an image rotation process to be performed and a trimming process of cutting out a partial image having a specified shape and size from the camera image after the image rotation process (1) to (5) The image processing apparatus according to any one of items 1 to 3.

(7) A camera image captured by a camera mounted on the observer's head together with a head-mounted display (HMD), and in an actual external environment observed by the observer together with the display image by the HMD An image processing method for performing image processing on an image of an object,
The image processing method is executed in at least the image display device of the image display device and the monitor device having the HMD,
The HMD projects image light representing an image according to an image signal onto an observer's eye, thereby displaying the image as a display image in a first image display area,
The monitor device displays the camera image as a monitor image in a second image display area based on camera image data representing the camera image received wirelessly or by wire from the image display device;
The image processing method is
An inclination angle detection step of detecting an inclination angle of the observer's head with respect to the direction of gravity and outputting inclination angle data representing the detected inclination angle;
Based on the output tilt angle data, image tilt correction for correcting the tilt of the image with respect to the gravitational direction in a direction opposite to the direction of the detected tilt angle and substantially the same angle is performed. An image inclination correction step that is performed on image data but not on display image data representing the display image, and
The image processing method, wherein the monitor device displays the monitor image in the second image display area based on corrected camera image data which is camera image data on which the image inclination correction has been performed.

(8) A program executed by a computer to implement the image processing method according to (7).

(9) A recording medium on which the program according to item (8) is recorded so as to be readable by a computer.

  According to the present invention, when the monitor device displays a camera image in which an object in the real outside world is imaged by a camera mounted on the observer's head together with the HMD to a person other than the observer. In addition, the image inclination correction is performed on the camera image. Therefore, according to the present invention, even if the observer tilts his / her head while the monitor image is displayed, the orientation of the object in the monitor image does not substantially change.

FIG. 1 is a plan view showing an appearance of an image display device among an image display device and a monitor device that jointly realize the image processing device according to the first embodiment of the present invention. It has a see-through HMD and a control unit. FIG. 2 is a functional block diagram showing the image display device shown in FIG. 1 together with the monitor device connected to the image display device via a network. The monitor device is connected to the image display device via a network. As a result, a remote support system is constructed. FIG. 3 is a front view showing an example of a support image and an example of a see-through image visually recognized by an observer in the entire image display area of the HMD shown in FIG. FIG. 4A shows a state in which the HMD shown in FIG. 1 is attached to the worker together with the camera and the camera image taken by the camera in the remote support system according to the first embodiment shown in FIG. FIG. 4B is a front view for explaining a state displayed on the screen of the apparatus. FIG. 4B is a diagram illustrating a conventional remote support system in which an object tilts in a monitor image when an operator tilts his / her neck. FIG. 4C is a front view for explaining the object, and in the remote support system according to the first embodiment, as an effect of the image tilt correction, the object is displayed in the monitor image even if the operator tilts the head. It is a front view for demonstrating that it does not need to incline. FIG. 5 is a flowchart conceptually showing a data transmission program executed by the CPU of the image display apparatus shown in FIG. FIG. 6 is a flowchart conceptually showing a reference inclination angle setting program executed by the CPU of the monitor apparatus shown in FIG. FIG. 7 is a flowchart conceptually showing an image display program executed by the CPU of the monitor apparatus shown in FIG. FIG. 8A is a front view for explaining an example of an image inclination correction algorithm in the first embodiment, and FIG. 8B shows another example of the image inclination correction algorithm. FIG. 8C is a front view for explaining still another example of the image inclination correction algorithm. FIG. 9 is a flowchart conceptually showing an image display program executed by the CPU of the image display apparatus shown in FIG. FIG. 10 shows examples of the support image and the check image observed together with the see-through image in the whole image display area of the HMD shown in FIG. It is a front view for demonstrating. FIG. 11 is a flowchart conceptually showing an image display program executed by the CPU of the image display device, out of the image display device and the monitor device that jointly realize the image processing device according to the second embodiment of the present invention. It is. FIG. 12 is a flowchart conceptually showing an image display program executed by the CPU of the monitor device in the second embodiment. FIG. 13 is a flowchart conceptually showing an image display program executed by the CPU of the image display device among the image display device and the monitor device that jointly realize the image processing device according to the third embodiment of the present invention. It is. FIG. 14 illustrates each example of the support image and the frame image observed together with the see-through image in the entire image display area of the image display device when the operator tilts his / her neck in the third embodiment. It is a front view for doing.

  Hereinafter, some of the exemplary embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a plan view showing an image display device 10 used for realizing the image processing device according to the first embodiment of the present invention. The image display device 10 includes a see-through type HMD 12. The HMD 12 is configured to project image light representing an image onto the eyes of an observer. The HMD 12 is attached to the head of the observer by the head mount device 14 while the observer is wearing spectacles (not shown).

  The head mount device 14 includes a frame 16 and an attachment device 18. Regardless of whether the frame 16 is a frame of eyeglasses (not shown) worn by the observer or a dedicated frame, the frame 16 is attached to the observer's head while being put on both ears of the observer. . The HMD 12 is mounted on a part of the frame 16 via the attachment device 18.

  Next, referring to FIGS. 1 and 2, the HMD 12 and the control unit 20 for controlling the HMD 12 will be described which are mainly composed of a computer. In the present embodiment, the HMD 12 and the control unit 20 which are both independent components are combined with each other to constitute the image display apparatus 10. However, the image display apparatus 10 is incorporated by incorporating the control unit 20 into the HMD 12. It may be configured.

  First, the outline of the HMD 12 will be described. FIG. 1 shows the HMD 12 in a plan view. The HMD 12 is a monocular system, and projects image light representing an image onto one eye of the observer to display the image to the observer. Further, the HMD 12 is a retinal scanning type, which scans a light beam having a strength corresponding to an image signal from a light source such as a laser by a scanner and projects the scanned light beam on the retina of an observer for observation. Enables a person to observe an image as a virtual image. Further, the HMD 12 is a see-through type, and allows an observer to observe a display image superimposed on the actual outside world.

  In addition, in this embodiment, the HMD 12 is a monocular type, a retinal scanning type, and a see-through type. However, the HMD 12 is changed to a binocular type, or planar light from a light source is changed to an LCD (liquid crystal display), an organic type Using spatial modulation elements such as EL (electroluminescence), each pixel is spatially modulated, and the modulated light is changed to a type that projects onto the retina of the observer, or in parallel with the display image observation. Thus, it is possible to change to a closed type in which the actual outside world cannot be observed.

  The present invention can be implemented using the HMD 12 regardless of whether the HMD 12 is a see-through type or a closed type. However, the HMD 12 uses the HMD 12 to enable the observer to observe the actual outside world together with the display image (for example, the support image shown in FIG. 3) displayed by the HMD 12. This is a necessary condition for carrying out the present invention.

  Therefore, when the HMD 12 is a closed type, the HMD 12 is a real external environment located in front of the observer, and if the HMD 12 is a see-through type, the see-through image ( For example, the same image as the see-through image shown in FIG. 5 captured by the camera 21 in the daytime or the dark field image captured by the infrared camera (not shown) at night is displayed to the observer as the second display image. It is necessary to configure so as to.

  In the present embodiment, a camera 21 (for example, a CCD (Charge-Coupled Device) camera) is fixed to the frame 16 in order to capture an actual outside world located in front of the observer. Thereby, the camera 21 is mounted on the observer's head indirectly through the frame 16. As a result, the camera 21 moves together with the HMD 12 in accordance with the movement of the observer's head. However, it goes without saying that the camera 21 may be directly mounted on the HMD 12, for example.

  Next, the control unit 20 will be described. A control unit 20 is connected to the HMD 12 via a cable 22 as shown in FIG. The cable 22 includes a control line that supplies a control signal, a power line that supplies electric power, and an optical fiber 82 (described in detail later) that transmits light. While the HMD 12 is mounted on the observer's head, the control unit 20 is mounted on a portion other than the head (for example, the waist) of the observer.

  The control unit 20 includes a CPU (Central Processing Unit) 26 as a processor, a program ROM (Read Only Memory) 27 and a flash ROM 28 as nonvolatile memories, And a RAM (Ramdom Access Memory) 29 as a memory.

  The control unit 20 further includes a power switch 30a operated by a user to turn on / off the power of the image display device 10, and a power source that is lit to indicate that the power of the image display device 10 is on. The lamp 30b and an operation unit (not shown, for example, a plurality of keys and buttons) are provided.

  The control unit 20 further includes a network interface (represented by “network I / F” in FIG. 2) 31a and a peripheral interface 31b (represented by “peripheral I / F” in FIG. 2). A video RAM 32 a, an HMD connection interface controller 32 b (represented by “HMD connection I / F controller” in FIG. 2), and a bus 33 are provided. The camera 21, the power switch 30a, and the power lamp 30b are connected to the peripheral interface 31b, and the HMD 12 is connected to the HMD connection interface controller 32b.

  The CPU 26, the program ROM 27, the flash ROM 28, the RAM 29, the network interface 31a, the peripheral interface 31b, the video RAM 32a, and the HMD connection interface controller 32b are connected to each other via the bus 33. Further, the network interface 31a and the video RAM 32a are directly connected to each other. Similarly, the video RAM 32a and the HMD connection interface controller 32b are directly connected to each other.

  An external device (not shown) such as a personal computer is connected to the peripheral interface 31b, and a video signal is input from the external device to the control unit 20 via the peripheral interface 31b. The video signal represents display content to be reproduced by the image display device 10. The input data representing the display content includes support image data representing a support image described later, and the support image data is stored in the flash ROM 28.

  The image light incident on the retina 124 allows the observer to observe the two-dimensional image as a virtual image. Not only the image light reflected by the half mirror 112 but also light from the external environment (external light) is transmitted through the half mirror 112 and incident on one eye of the observer. As a result, the observer can observe the actual outside world in parallel with the observation of the image displayed by the image light.

  In the example shown in FIG. 3, the HMD 12 displays an image corresponding to the image signal as a display image in the entire image display area (an example of the first image display area) 126. In this example, the display image is arranged at a predetermined position in the first image display area 126 and is displayed in the first sub-area 127 smaller than the first image display area 126. In this example, the display image is a support image displayed to the worker in order to support the work by the worker who is an observer, as described above. The worker observes the object 128 in the real outside world (in the example shown in FIG. 3, one tree extending straight from the ground) as a see-through image together with the support image. In addition to the HMD 12, a camera 21 is also mounted on the operator's head, and the camera 21 images the object 128 being observed by the operator together with the display image.

  In the present embodiment, the purpose of using the camera 21 is to perform work by a worker who is performing work while observing the object 128 while wearing the HMD 12 at a remote place from the worker. It is to enable a work supporter to assist. Specifically, the camera 21 images the object 128 so that the work supporter can monitor the object 128 being observed by the worker in a remote place in real time.

  In order to achieve the purpose of such remote support, as shown in FIG. 2, the image display device 10 having the HMD 12 used by the worker and the monitor device 130 used by the work supporter are wirelessly or wired. Are connected to each other via a network (for example, LAN) 140. In FIG. 4A, the worker is observing the object 128 with the HMD 12 and the camera 21 mounted on his / her head, and the work supporter in the remote place Conceptually shows that the object 128 being observed by the operator and captured by the camera 21 mounted on the operator's head is monitored in real time as a monitor image by the monitor device 130. Has been. According to this remote support system, it is possible for a person other than the observer to monitor the state of view of the observer and / or the state of the object in a remote place.

  As shown in FIG. 2, the monitor device 130 is configured mainly with a computer. Specifically, the monitor device 130 includes a CPU 142 as a processor, a ROM 144 as a nonvolatile memory, a RAM 146 as a volatile memory, and a bus 148 that connects these elements to each other.

  The bus 148 has a screen 150 (see FIG. 4A) having an image display area (an example of the second image display area) on which the monitor image is displayed, and a display for controlling the display operation of the screen 150. A display unit 160 having a controller (not shown) and an operation unit 162 operated by a work supporter to input data to the monitor device 130 are connected, and a network interface (in FIG. 2, “ 164 "is also connected. The network interface 164 is connected to the network 140, and the network interface 31 a in the image display apparatus 10 is also connected to the network 140. Thereby, the image display apparatus 10 and the monitor apparatus 130 can communicate with each other. As a result, a remote support system is constructed between the worker and the work supporter.

  As a result of research conducted by the inventor on the remote support system, it has been found that the conventional remote support system which does not take special measures has the following disadvantages. This will be specifically described with reference to FIG.

  When the operator tilts his / her head with the HMD 12 and the camera 21 mounted on his / her head, the HMD 12 and the camera 21 are also tilted together. As a result, the target in the camera image captured by the camera 21 The image of the object 128 is also tilted.

  Therefore, when the monitor device 130 is designed so that the camera image is displayed as it is as the monitor image, the object 128 itself is in an absolute space (with respect to the direction of gravity as shown in the center position in FIG. 4B). Although the image of the object 128 displayed on the screen 150 of the monitor device 130 is not inclined at all, the image of the object 128 is relative to the frame of the monitor image as shown on the right side in FIG. Therefore, the operator tilts in the direction opposite to the direction in which the operator tilts the neck and at the same angle as the tilt angle of the neck. Since the act of tilting the head frequently occurs even if there is a difference in tilt angle, the work supporter repeatedly tilts the object 128 that is not actually tilted left and right on the screen 150. Observe in the state.

  By the way, in order to express a specific direction, the term which defines a direction on absolute space may be used. Examples of such terms include up, down, left, right, portrait, landscape, vertical, and horizontal. Therefore, in a state where the target object 128 is displayed in an inclined state on the screen 150, the work support person uses a term that defines a specific direction for the tree that is the target object 128 in the absolute space, Consider when to indicate.

  As an example, if the work supporter does not use the direction defined in the absolute space, it is necessary to indicate the direction to be expressed as "the direction in which the tree is extended from the tip of the tree" Is assumed. In this case, the work supporter uses the term “upward”, which is a term defined in absolute space, while looking at the monitor image in which the tree is displayed with an inclination, for example, “the tip of the tree” There is a possibility that it is expressed as “left diagonally upward”.

  In this case, when the worker listens to the expression, the worker may interpret that the expression means “diagonally left upward from the tip of the tree extending in the direction of gravity in the real world”. This interpretation does not coincide with the above-mentioned “direction in which the tree extends from the tip of the tree”. For this reason, there arises a disadvantage that the instruction regarding the direction of the work supporter is not correctly understood by the worker.

  In order to prevent such a discrepancy in recognition of direction between the work supporter and the worker, in this embodiment, as shown in FIG. An inclination angle with respect to the direction is detected, and the inclination of the camera image obtained by imaging the object 128 with respect to the direction of gravity is opposite to the direction of the inclination angle and substantially the same according to the detected inclination angle. Image tilt correction that is corrected by angle is performed. However, this image inclination correction is not performed on the display image (the above-described support image) by the HMD 12.

  When the above-described image inclination correction is performed, a monitor image to be displayed on the screen 150 of the monitor device 130 is generated with respect to the camera image obtained by imaging the object 128, and as a result, the operator's head is displayed. Regardless of inclination, the object 128 is displayed on the screen 150 so as to have the same orientation as the actual orientation in the absolute space. In the example shown in FIG. 4, the object 128 is always displayed on the screen 150 in a posture extending in the direction of gravity regardless of whether the operator's head is inclined.

  As a result, the orientation of the object 128 recognized by the worker and the orientation of the object 128 that the work support person observes on the screen 130 (orientation in the absolute space) always coincide with each other. Thus, inconsistency in recognition of direction between the work supporter and the worker is prevented.

  Further, when the above-described image tilt correction is performed, the target 128 always has the same direction on the screen 150 regardless of whether the head of the operator is tilted or not, unless the actual direction of the target 128 changes. The orientation of the object 128 in the monitor image on the screen 150 is stabilized.

  As described above, in order to perform the image tilt correction schematically described above, in the present embodiment, as shown in FIG. 2, the tilt angle sensor 170 is mounted on the head of the worker wearing the HMD 12, and Some programs for correcting the image tilt are stored in advance in the program ROM 27 of the image display device 10 and the ROM 144 of the monitor device 130. The tilt angle sensor 170 is electrically connected to the peripheral interface 31b.

  The tilt angle sensor 170 is attached to the worker's head in order to detect the tilt direction (rightward or leftward) and angle of the worker's head. The tilt angle sensor 170 may be directly attached to the operator's head, but is directly attached to another member that is directly attached to the head, such as the HMD 12, the frame 16, the camera 21, or the like. May be. The tilt angle sensor 170 detects the rotation of the operator's head around an axis extending in the front-rear direction. The tilt angle sensor 170 can be mainly composed of a gyro sensor, for example. In this case, it is possible to calculate the tilt angle of the head by time-integrating the angular velocity detected by the gyro sensor.

  Next, image processing for correcting the image tilt will be described in detail.

  FIG. 5 conceptually shows a data transmission program executed by the CPU 26 of the image display apparatus 10 in a flowchart. This data transmission program is repeatedly executed while the image display apparatus 10 is powered on. At the time of execution of each time, first, in step S1, camera image data (pre-correction camera image data) representing the imaging result thereof is taken into the CPU 26 from the camera 21. Next, in step S2, the captured camera image data is transmitted from the network interface 31a to the monitor device 130 via the network 140.

  Subsequently, in step S3, the sensor signal from the tilt angle sensor 170 is taken into the CPU 26, and the tilt angle θ of the operator's head is detected based on the sensor signal. Thereafter, in step S4, tilt angle data representing the detected tilt angle θ is transmitted from the network interface 31a to the monitor device 130 via the network 140. Thus, one execution of this data transmission program is completed.

  In FIG. 6, a reference inclination angle setting program executed by the CPU 142 of the monitor device 130 is conceptually represented by a flowchart. This reference inclination angle setting program is repeatedly executed while the monitor device 130 is powered on.

  The work supporter performs a calibration work for the head inclination angle detection value of the worker prior to each remote support work. Specifically, the work supporter can determine the worker's direction from the direction of the actual outside world currently displayed on the screen 150 (for example, the direction of the object 128) based on the camera image data received from the image display device 10. It is determined whether or not the camera 21 mounted on the head is positioned on a horizontal plane (for example, whether or not the object 128 in the camera image extends vertically on the screen 150). If it is determined that the camera 21 is positioned on the horizontal plane, the work supporter performs a horizontal reset operation via the operation unit 162.

  It should be noted that the same calibration work can be performed by the worker wearing the camera 21 by himself instead of the work supporter. For example, in the image display device 10, the angular velocity detected by the gyro sensor is time-integrated from the reference time, that is, the time when the operator's head is in the upright position from the time when the operator has input. It is possible to calculate the tilt angle of the head as a relative angle with respect to the direction of gravity (equal to an absolute angle in absolute space).

  However, even if the camera 21 is mounted on the worker's head while being tilted, the worker may not be able to notice it. In this case, the worker cannot perform the calibration work described above with high accuracy.

  In contrast, the work supporter uses the camera image that reflects not only the inclination of the head of the worker but also the mounting angle of the camera 21 with respect to the worker, and the actual external environment that the worker observes. Can be confirmed on the screen 150. Therefore, the work supporter can perform the above-described calibration work with high accuracy.

  When the reference inclination angle setting program shown in FIG. 6 is executed each time, first, in step S101, it is determined whether or not the work support person has performed the horizontal reset operation on the operation unit 162. In this case, assuming that the work supporter has not performed the horizontal reset operation, one execution of the reference inclination angle setting program is immediately terminated.

  On the other hand, this time, assuming that the work supporter has performed a horizontal reset operation, the latest tilt angle data is received from the image display device 10 in step S102. Subsequently, in step S103, the inclination angle θ represented by the received inclination angle data is set to the reference inclination angle θREF. The set reference inclination angle θREF is stored in the RAM 146 for later use. The difference between the tilt angle θ and the set tilt angle θREF represents the tilt angle of the worker's head from the direction of gravity in the absolute space (hereinafter referred to as “absolute tilt angle θabs”). Thus, one execution of the reference inclination angle setting program is completed.

  FIG. 7 conceptually shows a flowchart of an image display program executed by the CPU 142 of the monitor device 130. This image display program is repeatedly executed while the power of the monitor device 130 is on.

  At the time of each execution, first, in step S201, the latest camera image data and the latest tilt angle data are received from the image display device 10. Next, in step S202, whether or not the current inclination angle θ represented by the received inclination angle data is deviated from the reference inclination angle θREF, that is, whether or not the absolute inclination angle θabs is zero. Is determined. Thereby, it is determined whether or not the operator's head is inclined with respect to the direction of gravity, and as a result, whether or not the object 128 in the camera image is inclined with respect to the frame of the camera image. Is determined.

  This time, assuming that the current inclination angle θ is not deviated from the reference inclination angle θREF, monitor image data representing a monitor image to be displayed on the screen 150 without performing the image inclination correction in step S203. Are generated to represent the same image as the color image data.

  In contrast, assuming that the current inclination angle θ is deviated from the reference inclination angle θREF this time, the image inclination correction is performed on the color image data in step S204. Specifically, as shown in FIG. 8A, first, an original camera image that is a camera image before image tilt correction is set to a position that matches a specified rotation center point (for example, the center point of the original camera image). ) Around the image is rotated in the direction opposite to the absolute inclination angle θabs and substantially the same as the absolute inclination angle θabs. FIG. 8A shows the original camera image after the image rotation process. At this stage, the object 128 in the image extends in the direction of gravity even though the absolute inclination angle θabs of the operator's head is not zero.

  Next, a trimming process is performed to cut out a partial image having a specified shape and size from the original camera image after the image rotation process. The partial image is positioned with respect to the original camera image such that its center point coincides with the center point of the original camera image. By this trimming process, a monitor image to be displayed on the screen 150 as a whole is generated. The screen 150 and the monitor image (rectangular image) to be displayed on the screen 150 have the same size and shape.

  Thereafter, in step S205, monitor image data representing the generated monitor image is generated.

  Regardless of whether step S203 is executed or step S205 is executed, a monitor image is then displayed on the screen 150 in step S206 based on the generated monitor image data.

  Subsequently, in step S207, the operator sends a request (a check image display request described later) for displaying the same image as the monitor image on the HMD 12 as a signal transmitted from the image display apparatus 10 to the image display apparatus 10. It is determined whether or not it represents the input. If the operator has not input the check image display request, one execution of the image display program is immediately terminated. If the operator has input the check image display request, step S208 is performed. The monitor image data is transmitted to the image display device 10 in order to display the check image on the HMD 12. This completes one execution of the image display program.

  Here, with reference to FIG. 8, some examples of the image tilt correction algorithm will be described. In the example of the algorithm shown in FIG. 8A, as described above, the camera image is rotated in a state where the center point of the monitor image coincides with the center point of the camera image, and the rotation angle is not so large. In combination with the fact that the size of the monitor image is not so large, the monitor image does not protrude from the rotated camera image.

  On the other hand, in the example of the algorithm shown in FIG. 8B, the center point of the monitor image is moved from the center point of the camera image by the operation of the work supporter, so that a part of the monitor image is obtained. It protrudes from the original camera image. The portion of the monitor image that protrudes from the original camera image is a portion where no camera image data exists, and this portion cannot be displayed on the screen 150 unless special measures are taken.

  Therefore, in this algorithm example, monitor image data is generated so that the portion of the monitor image that protrudes from the original camera image is colored black. Thereby, the work supporter can know that the monitor image includes a portion that protrudes from the field of view of the camera 21.

  In the example of the algorithm shown in FIG. 8A, since the angle at which the camera image is rotated with respect to the monitor image is small, the monitor image does not protrude from the rotated camera image, but the angle is large. In such a case, the monitor image protrudes from the rotated camera image. On the other hand, in the example of the algorithm shown in FIG. 8C, the size of the monitor image is set according to the rotation angle of the camera image so that the monitor image has a maximum area that does not protrude from the rotated camera image. Can be stretched.

  FIG. 9 conceptually shows a flowchart of an image display program executed by the CPU 26 of the image display device 10. This image display program is repeatedly executed while the image display apparatus 10 is powered on. At each execution, first, in step S301, support image data representing a support image (see FIG. 10) to be displayed in the first sub-area 127 is read from the flash ROM 27.

  Next, in step S302, the operator inputs a check image display request for displaying the check image (see FIG. 10) in the second sub area 180 together with the support image to the image display device 10. It is determined whether or not. The second sub-area 180 is a predetermined position in the entire image display area 126, is visually distinguished from the first sub-area 127, and is arranged at a position that does not overlap therewith.

  If the operator has not input the check image display request, display image data is generated in step S303 using the read support image data as it is.

  On the other hand, when the operator inputs the check image display request, the monitor image data is received from the monitor device 130 in step S304. Subsequently, in step S305, based on the received monitor image data, the same monitor image currently being observed by the work supporter is displayed in the second subarea 180 in real time as the check image in a reduced state. To generate the check image data.

  Thereafter, in step S306, display image data is generated by synthesizing the read support image data and the generated check image data. As shown in FIG. 10, the generated display image data is used to display the support image in the first subarea 126 and display the check image in the second subarea 180 together. It is data of.

  Regardless of whether step S303 is executed or step S306 is executed, after that, in step S307, based on the generated display image data, a display signal (for example, HMD12) for displaying an image on the HMD 12 is displayed. Is a signal for controlling a light source such as a laser (for example, a luminance signal or an intensity signal), and when the HMD 12 is the above-described spatial modulation type, an LCD is used. , A signal (for example, a pixel signal) for controlling the spatial modulation element such as an organic EL is generated. Subsequently, in step S308, the generated display signal is output to the light source or the spatial modulation element. Thereby, as shown in FIG. 10, when the support image is displayed in the first sub-area 126 and the operator inputs the check image display request, the check image is also displayed in the second sub-area. 180 is displayed. This completes one execution of the image display program.

  Thus, the operator can observe the support image and the check image while observing the object 128 in the actual outside world as a see-through image. By observing the check image, the operator can monitor the monitor image that the work supporter is observing (for example, an image that is the basis for the instruction when the work support person instructs the worker in a specific direction). The same image can be confirmed in real time in a reduced state.

  Next, a second embodiment of the present invention will be described. However, since this embodiment has many elements in common with the first embodiment, only different elements will be described in detail, and the common elements will be duplicated by quoting using the same symbols or names. Description is omitted.

  In the first embodiment, the image processing called the image inclination correction is executed in the monitor device 130. On the other hand, in the present embodiment, the same image processing is performed in the image display device 10.

  In the present embodiment, the CPU 26 of the image display device 10 executes the image display program conceptually shown in the flowchart in FIG. 11 in place of the data transmission program shown in FIG. 5 and the image display program shown in FIG. Is done. Furthermore, instead of the reference tilt angle setting program shown in FIG. 6 being executed by the CPU 142 of the monitor device 130, a program substantially the same as the reference tilt angle setting program shown in FIG. 6 is executed by the CPU 26 of the image display device 10. The

  When the image display program shown in FIG. 11 is executed each time, first, in step S401, the support image data is read from the flash ROM 28 in the same manner as in step S301 shown in FIG. 9, and then in step S402, FIG. The tilt angle θ is detected in the same manner as in step S3. Subsequently, in step S403, camera image data is captured from the camera 21 in the same manner as in step S1 shown in FIG.

  Thereafter, steps S404 to S407 are executed in the same manner as steps S202 to S205 shown in FIG. Subsequently, in step S <b> 408, monitor image data formed by executing step S <b> 405 or S <b> 407 is transmitted to the monitor device 130.

  Subsequently, in step S409, similarly to step S302 shown in FIG. 9, it is determined whether or not the operator has input the check image display request to the image display device 10. If the operator has not input the check image display request, display image data is generated in step S410 using the read support image data as it is, as in step S303 shown in FIG. The

  On the other hand, when the operator inputs the check image display request, in step S411, as the check image data, based on the monitor image data captured in step S403 in accordance with step S305 shown in FIG. Generated. Thereafter, in step S412, the display image data is generated by synthesizing the support image data read in step S401 and the check image data generated in step S411 similarly to step S306 shown in FIG. The

  Regardless of whether step S410 is executed or step S412 is executed, in step S413, the display signal is generated based on the generated display image data in the same manner as step S307 shown in FIG. Is generated. Subsequently, in step S414, as in step S308 shown in FIG. 9, the generated display signal is output to the light source or the spatial modulation element, so that a support image is displayed in the first sub-area 126. When the operator inputs the check image display request, a check image is also displayed in the second sub area 180. This completes one execution of the image display program.

  Further, in the present embodiment, the CPU 142 of the monitor device 130 executes the image display program conceptually shown in the flowchart in FIG. 12 instead of the image display program shown in FIG.

  When the image display program shown in FIG. 11 is executed each time, first, the monitor image data is received from the image display device 10 in step S501. In step S502, the monitor image is displayed on the screen 150 based on the received monitor image data. This completes one execution of the image display program.

  Next, a third embodiment of the present invention will be described. However, since this embodiment has many elements in common with the second embodiment, only different elements will be described in detail, and the common elements will be duplicated by quoting using the same reference numerals or names. Description is omitted.

  In the second embodiment, the same monitor image that the work supporter is observing on the monitor device 130 is displayed by the HMD 12 as a check image, whereby the worker is observing the monitor image that the work support person is observing. It was possible to confirm.

  On the other hand, in the present embodiment, as shown in FIG. 14, the image display apparatus 10 displays the whole image (the support image (virtual image) and the see-through image (real image) that the operator observes in the whole image display area 126. ))), The matching partial image 190 that matches the monitor image (rectangular image including the image of the object 128 and the background image thereof) observed by the work support person on the monitor device 130 is matched. A frame image 200 (an image representing a rectangular frame surrounding the matching partial image 190) for visually distinguishing from the non-matching partial image 192 is designed to be displayed in the entire image display area 126.

  In the present embodiment, the CPU 26 of the image display apparatus 10 executes an image display program conceptually shown in a flowchart in FIG. 13 instead of the image display program shown in FIG.

  At each execution of the image display program shown in FIG. 13, steps S501 to S508 are executed in the same manner as S401 to S408 shown in FIG.

  Thereafter, in step S509, it is determined whether or not the operator has input a frame image display request for displaying the frame image 200 in the entire image display area 126 to the image display device 10. If the operator has not input the frame image display request, display image data is generated in step S410 using the read support image data as it is.

  On the other hand, when the worker inputs the frame image display request, in step S411, frame image data representing the frame image 200 is generated based on the generated monitor image data.

  The details of step S411 will be described. In the image display apparatus 10, the relative position between the center positions of the entire image display area 126 and the center position of the camera image is matched, for example, such that the center position of the camera image coincides with each other. The position is known. Further, the relative angle between the entire image display area 126 and the camera image is also known, such as 0. Further, the relationship between the size and the shape between the whole image display area 126 and the camera image is known as well.

  That is, in the image display device 10, the relationship regarding the relative position, the relative angle, the size, and the shape between the entire image display area 126 (the entire image) and the camera image is known. It is possible to uniquely derive the geometric information of the camera image from the geometric information.

  Further, in the image display device 10, as a result of the execution of step S506, the relationship regarding the relative position, relative angle, size, and shape between the monitor image and the camera image is known. Therefore, in the image display device 10, the geometric information of the monitor image (including the geometric information related to the frame of the monitor image) can be uniquely derived from the geometric information of the entire image display area 126. It is.

  Therefore, in this step S411, a matching partial image 190 that matches the monitor image is specified from the whole image in the whole image display area 126 based on the generated monitor image data. The geometric information of the frame image 200 representing the frame is identified. The thickness and line type of the frame image 200 are known. Then, frame image data representing the frame image 200 is generated.

  After that, in step S512, display image data is generated by synthesizing the read support image data and the generated frame image data. As shown in FIG. 14, the generated display image data displays the support image in the first sub area 126 and displays the frame image 200 in the entire image display area 126 together. It is data for.

  Regardless of whether step S510 is executed or step S512 is executed, the display signal is then generated in step S513 based on the generated display image data. Subsequently, in step S514, the generated display signal is output to the light source or the spatial modulation element. As a result, as shown in FIG. 14, when the support image is displayed in the first sub-area 126 and the operator inputs the frame image display request, the frame image 200 is also displayed as the entire image. It is displayed in the area 126. This completes one execution of the image display program.

  Thus, the operator observes the support image while observing the object 128 in the real outside world as a see-through image, and further, the same monitor image that the work supporter observes on the monitor device 130 (monitor image) Can be observed in real time in consideration of the orientation of the frame image 200. Thereby, the worker can recognize in common with the work supporter the relative orientation of the object 128 in the monitor image observed by the work supporter with respect to the frame of the monitor image.

  As is clear from the above description, in the present embodiment, for convenience of explanation, the tilt angle sensor 170 and the portion of the image display device 10 that executes step S3 shown in FIG. The “tilt angle detection unit” in the item (1) is configured as an example, and the image display device 10 executes steps S1, S2, and S4 shown in FIG. An example of the “image inclination correction unit” in the same section is obtained by jointly executing a part for executing the reference inclination angle setting program shown in FIG. 6 and a part for executing steps S302 to S306 shown in FIG. It is possible to think that it constitutes.

  Furthermore, in the present embodiment, for convenience of explanation, the network I / F 31a and the part of the image display device 10 that executes steps S1, S2, and S4 shown in FIG. 7 constitutes an example of the “transmission unit”, and the part of the monitor device 130 that executes steps S201, S202, S204, and S205 shown in FIG. 7 is an example of the “image tilt correction unit” in the same term. It is possible to think that it is composed.

  Further, in the present embodiment, for convenience of explanation, the portion of the image display device 10 that executes steps S402 to S404, S406, and S407 shown in FIG. 11 is the “image inclination correction unit” in the above (3). An example is configured, and the network I / F 31a and the part of the image display device 10 that executes step S408 illustrated in FIG. 11 cooperate with each other to configure an example of “transmission unit” in the same section. It is possible to think that

  Furthermore, in the present embodiment, for the sake of convenience of explanation, the part that executes steps S302 and S304 to S306 shown in FIG. 9 in the image display device 10 is an example of the “check image display unit” in the above section (4). The portion that executes steps S502 to S504, S506, S507, and S509 to S512 shown in FIG. 13 in the image display device 10 constitutes an example of the “frame image display unit” in the section (5). It is possible to think that

  As mentioned above, although several embodiment of this invention was described in detail based on drawing, these are illustrations and are based on the knowledge of those skilled in the art including the aspect as described in the above-mentioned [summary of invention] column. The present invention can be implemented in other forms with various modifications and improvements.

Claims (6)

A camera image captured by a camera mounted on an observer's head together with a head-mounted display (HMD), and an object in the real outside world that the observer observes together with the display image by the HMD An image processing apparatus that performs image processing on a captured image,
The image processing device is provided in at least the image display device of the image display device and the monitor device having the HMD,
The HMD projects image light representing an image according to an image signal onto an observer's eye, thereby displaying the image as a display image in a first image display area,
The monitor device displays the camera image as a monitor image in a second image display area based on camera image data representing the camera image received wirelessly or by wire from the image display device;
The image processing apparatus
An inclination angle detector that detects an inclination angle of the head of the observer with respect to the direction of gravity and outputs inclination angle data representing the detected inclination angle;
Based on the output tilt angle data, image tilt correction for correcting the tilt of the image with respect to the gravitational direction in a direction opposite to the direction of the detected tilt angle and substantially the same angle is performed. An image tilt correction unit that performs image data but does not perform display image data representing the display image,
The monitor device displays the monitor image in the second image display area based on corrected camera image data which is camera image data on which the image tilt correction has been performed. further,
A transmission unit provided in the image display device, for transmitting the camera image data and the tilt angle data to the monitor device;
The said image inclination correction | amendment part is provided in the said monitor apparatus, Based on the inclination angle data received from the said transmission part, the said image inclination correction | amendment is performed with respect to the camera image data received from the said transmission part. Image processing device. The image tilt correction unit is provided in the image display device, performs the image tilt correction on the camera image data based on the tilt angle data,
The image processing apparatus further includes:
A transmission unit that is provided in the image display device and transmits the corrected camera image data to the monitor device;
The image processing device according to claim 1, wherein the monitor device displays the monitor image in the second image display area based on the corrected camera image data received from the transmission unit. further,
And a check image display unit that is provided in the image display device and displays the corrected camera image as a check image for an observer in a sub-area arranged at a predetermined position in the first image display area. Item 4. The image processing device according to any one of Items 1 to 3. further,
The image display device is a see-through type in which external light representing the object is projected onto the first image display area together with display light representing the display image, or the object is imaged by the camera. Regardless of whether the display light representing both the captured image and the display image is projected onto the first image display area, the observer observes the entire image in the first image display area. Among the images including the display image and the image of the object, a frame image for visually distinguishing a matched partial image that matches the corrected camera image from a mismatched partial image is the first image. The image processing apparatus according to claim 1, further comprising a frame image display unit that displays in the image display area.   The image tilt correction unit rotates the camera image before the image tilt correction around the rotation center point where the position is specified, in an opposite direction to the tilt angle and at substantially the same angle. 6. The corrected camera image is acquired by performing a process and a trimming process of cutting out a partial image having a specified shape and size from the camera image after the image rotation process. The image processing apparatus described.

Images