JP5683402B2 - Image composition apparatus and image composition method - Google Patents

Description

  The present invention relates to an image composition device that displays a composite image in which, for example, a real-world image and a CG image are superimposed on a transmissive display device such as an HMD (Head Mounted Display) according to the line-of-sight direction of the operator, The present invention relates to an image composition method.

For example, in the medical field, an image composition device or the like that displays a composite image in which additional information related to surgery (for example, information indicating the procedure of surgery) is superimposed on a surgical scene image on a transmissive display device is assumed. .
Also, in the industrial field, an image composition device or the like that displays a composite image in which information indicating work content or the like is superimposed on an image of an assembly view of an industrial product is displayed on a transmissive display device.

At this time, since a general image composition device does not have a function of designating a region to superimpose a CG image when superimposing a CG image on a real-world image, the CG image is displayed in a region that is being watched by the operator. May be superposed, making it difficult to see the video in the area of interest.
Patent Document 1 below discloses an image composition device that includes means for setting a region of interest to an operator as a non-display region, and prevents a CG image from being superimposed in the non-display region.
For example, a region that affects the work of the operator (for example, a region where instruments exist) is assumed as a region that the operator pays attention to, and a CG image is not superimposed on such a region.

JP 2004-178554 A (paragraph number [0009])

  Since the conventional image synthesizing apparatus is configured as described above, the CG image is not superimposed on the set non-display area, and it is possible to prevent the CG image from becoming difficult to see the video of the area of interest. can do. However, there is a problem that even if an object that the operator needs to recognize exists in the non-display area, the operator cannot be notified of the presence of the object.

  The present invention has been made in order to solve the above-described problems. If an object that needs to be recognized by the operator is present in the non-display area, the image composition can notify the operator to that effect. An object is to obtain an apparatus and an image composition method.

An image composition device according to the present invention includes position line-of-sight information acquisition means for acquiring position line-of-sight information indicating the position and line-of-sight direction of an operator, and a non-display area for setting a non-display area that does not display an image in a transmissive display device Display image generation for generating a display image in the line-of-sight direction indicated by the position line-of-sight information from the stored display image data with the viewpoint of the operator's position indicated by the position line-of-sight information acquired by the setting means and the position line-of-sight information acquisition means And if there is an object that the operator needs to recognize in the image in the non-display area in the display image generated by the display image generating means, the fact that the object is present is notified. CG image generation means for generating a CG image is provided, and the image composition means deletes the image in the non-display area from the display image generated by the display image generation means, and the CG image generation means If a CG image is not generated, a display image from which the image in the non-display area is deleted is displayed on the transmissive display device. If a CG image is generated by the CG image generation means, the image in the non-display area is displayed. The CG image is synthesized in an area other than the non-display area in the deleted display image, and the synthesized image is displayed on the transmissive display device.

According to this invention, the position line-of-sight information acquisition means for acquiring position line-of-sight information indicating the position and line-of-sight direction of the operator, and the non-display area setting means for setting a non-display area that does not display an image in the transmissive display device Display image generation means for generating a display image in the line-of-sight direction indicated by the position line-of-sight information from the stored display image data, with the position of the operator indicated by the position-line-of-sight information acquired by the position line-of-sight information acquisition means as a viewpoint; If there is an object that needs to be recognized by the operator among the images in the non-display area in the display image generated by the display image generating means, a CG image notifying that the object is present is displayed. CG image generating means for generating the image, and the image synthesizing means deletes the image in the non-display area from the display image generated by the display image generating means, and the CG image generating means generates the CG image. If not, the display image from which the image in the non-display area is deleted is displayed on the transmissive display device, and if the CG image is generated by the CG image generating means, the display in which the image in the non-display area is deleted is displayed. Since the CG image is synthesized with an area other than the non-display area in the image and the synthesized image is displayed on the transmissive display device, an object that the operator needs to recognize exists in the non-display area. If so, the operator can be notified of this.

It is a block diagram which shows the image synthesizing | combining apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (image composition method) of the image composition apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows an example when the object which an operator needs to recognize exists in a non-display area | region. It is explanatory drawing which shows the example which the object which an operator needs to recognize exists in a non-display area | region. It is explanatory drawing which shows an example of the image of the circle centering on the position where the object exists. It is explanatory drawing which shows an example of the image of the some straight line which cross | intersects in the position where the object exists. It is explanatory drawing which shows an example of the CG image which can grasp | ascertain the magnitude | size of an object.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an image composition apparatus according to Embodiment 1 of the present invention.
In FIG. 1, the transmissive display device 1 corresponds to, for example, an HMD (Head Mounted Display), and is a display device that displays a composite image generated by the image composition device 4.
The position / posture measuring device 2 is built in the transmissive display device 1, detects the operator's position and line-of-sight direction, and outputs position line-of-sight information indicating the operator's position and line-of-sight direction to the image composition device 4. Device.
The position / orientation measurement apparatus 2 includes, for example, a geomagnetic sensor, a gyroscope, an optical sensor, and the like.

The display image data storage unit 3 is composed of a storage device such as a RAM or a hard disk, for example. The display image data used when the image composition device 4 generates a display image (for example, CG three-dimensional data, a plurality of cameras). All-around image data centered on the photographed operator).
The image composition device 4 is a device that generates a display image in the line of sight of the operator and displays the display image on the transmissive display device 1.

  The position line-of-sight information acquisition unit 11 of the image composition device 4 includes an interface (for example, a USB port) for acquiring information from the outside, and performs a process of acquiring the position line-of-sight information output from the position / posture measurement apparatus 2. . The position line-of-sight information acquisition unit 11 constitutes position line-of-sight information acquisition means.

The non-display area setting unit 12 includes, for example, a man-machine interface such as a keyboard and a mouse. The non-display area setting unit 12 accepts the setting of a non-display area that does not display an image in the transparent display device 1 according to the setting operation by the operator. A process of outputting coordinate information indicating the position of the display area to the non-display area calculation unit 14 is performed. The non-display area setting unit 12 constitutes non-display area setting means.
In the first embodiment, it is assumed that the non-display area setting unit 12 is configured by a man-machine interface such as a keyboard and a mouse. However, a communication interface that receives setting information transmitted from the outside, etc. In this case, in accordance with the setting information, a non-display area setting process in which no image is displayed is performed in the transmissive display device 1.

  The display image generation unit 13 includes, for example, a GPU (Graphics Processing Unit), and the display image data storage unit 3 uses the position of the operator indicated by the position line-of-sight information acquired by the position line-of-sight information acquisition unit 11 as a viewpoint. A process of generating a display image in the line-of-sight direction indicated by the position line-of-sight information from the accumulated display image data is performed. The display image generation unit 13 constitutes a display image generation unit.

The non-display area calculation unit 14 includes, for example, a semiconductor integrated circuit on which a CPU is mounted, a one-chip microcomputer, or the like. The position line-of-sight information acquired by the position-line-of-sight information acquisition unit 11 and the non-display area setting unit 12 The non-display area of the transmissive display device 1 is calculated from the coordinate information set by the above.
The non-display area information retrieval unit 15 is configured by, for example, a semiconductor integrated circuit on which a CPU is mounted, a one-chip microcomputer, or the like, and an image in the non-display area in the display image generated by the display image generation unit 13. A process for determining whether or not there is an object that the operator needs to recognize is performed.

The CG image generation unit 16 is composed of, for example, a GPU. If the non-display area information search unit 15 determines that an object that the operator needs to recognize exists, the object exists. The process which produces | generates the CG image which notifies that there exists is implemented.
The non-display area calculation unit 14, the non-display area information search unit 15, and the CG image generation unit 16 constitute a CG image generation unit.

  The image composition unit 17 is configured by, for example, a GPU, a semiconductor integrated circuit on which a CPU is mounted, or a one-chip microcomputer, and is calculated by the non-display area calculation unit 14 from the display image generated by the display image generation unit 13. If the image in the non-display area is deleted, and the CG image generator 16 does not generate a CG image, the display image from which the image in the non-display area is deleted is displayed on the transmissive display device 1. To do. If the CG image is generated by the CG image generation unit 16, the CG image is combined with the display image from which the image in the non-display area is deleted, and the combined image is displayed on the transmissive display device 1. . The image composition unit 17 constitutes an image composition unit.

In the example of FIG. 1, the position line-of-sight information acquisition unit 11, the non-display area setting unit 12, the display image generation unit 13, the non-display area calculation unit 14, and the non-display area internal information search unit 15 that are components of the image composition device 4. Although it is assumed that each of the CG image generation unit 16 and the image synthesis unit 17 is configured by dedicated hardware, all or part of the image synthesis device 4 may be configured by a computer.
When the entire image composition device 4 is configured by a computer, the position / gaze information acquisition unit 11, the non-display area setting unit 12, the display image generation unit 13, the non-display area calculation unit 14, and the non-display area information search unit 15. The program describing the processing contents of the CG image generation unit 16 and the image composition unit 17 may be stored in the memory of a computer, and the CPU of the computer may execute the program stored in the memory.
FIG. 2 is a flowchart showing the processing contents (image composition method) of the image composition apparatus according to Embodiment 1 of the present invention.

Next, the operation will be described.
The position / posture measurement apparatus 2 detects the operator's position and line-of-sight direction, and outputs position line-of-sight information indicating the operator's position and line-of-sight direction to the image composition apparatus 4.
Since the position and gaze direction detection processing itself in the position / orientation measurement apparatus 2 is a known technique, a detailed description thereof will be omitted.
The position line-of-sight information acquisition unit 11 of the image composition apparatus 4 acquires the position line-of-sight information output from the position / posture measurement apparatus 2 and outputs the position line-of-sight information to the display image generation unit 13 and the non-display area calculation unit 14. (Step ST1).

For example, the non-display area setting unit 12 accepts the setting of a non-display area where no image is displayed in the transmissive display device 1 in accordance with the setting operation of the operator, and calculates coordinate information indicating the position of the non-display area as a non-display area calculation. It outputs to the part 14 (step ST2).
In the example of FIG. 2, after the position / line-of-sight information acquisition unit 11 acquires the position / line-of-sight information output from the position / posture measurement apparatus 2, the non-display area setting unit 12 accepts the setting of the non-display area. However, before the position / line-of-sight information acquisition unit 11 acquires the position / line-of-sight information output from the position / orientation measurement device 2, the non-display area setting unit 12 may accept the setting of the non-display area. Good.

When receiving the position line-of-sight information from the position line-of-sight information acquisition unit 11, the display image generation unit 13 uses the position of the operator indicated by the position line-of-sight information as a viewpoint from the display image data accumulated by the display image data accumulation unit 3. Then, a display image in the line-of-sight direction indicated by the position line-of-sight information is generated (step ST3).
The processing itself for generating a display image in the line-of-sight direction with the operator's position as the viewpoint is a known technique. For example, if the display image data stored in the display image data storage unit 3 is CG three-dimensional data. For example, by setting the operator's position and line-of-sight direction in the CG three-dimensional data, a display image in the line-of-sight direction with the operator's position as the viewpoint can be generated.
In addition, if the display image data stored by the display image data storage unit 3 is all-around image data centered on an operator photographed by a plurality of cameras, the entire periphery of the operator's position is used as a viewpoint. By cutting out the image in the line of sight from the image data, it is possible to generate a display image in the line of sight with the position of the operator as the viewpoint.

When the non-display area calculation unit 14 receives the position line-of-sight information from the position line-of-sight information acquisition unit 11 and the coordinate information from the non-display area setting unit 12, the non-display area calculation unit 14 uses the position line-of-sight information and the coordinate information. The display area is calculated.
For example, the position and line-of-sight direction of the operator when the non-display area setting unit 12 sets the non-display area (the position and line-of-sight direction of the operator when the non-display area is set are determined from the position / posture measurement device 2, for example. Can be grasped by acquiring the position line-of-sight information measured at the time of setting the display area) is the same as the operator's position and line-of-sight direction indicated by the position line-of-sight information output from the position line-of-sight information acquisition unit 11 (When the position and line-of-sight direction of the operator at the time of setting the non-display area coincides with the position and line-of-sight direction at the time of measurement by the position / posture measurement device 2) Since the position indicated by the coordinate information is the position of the non-display area of the transmissive display device 1, the position indicated by the coordinate information is set as the position indicating the non-display area of the transmissive display device 1. And image composition unit And outputs it to the 7.

On the other hand, the position and line-of-sight direction of the operator when the non-display area setting unit 12 sets the non-display area are different from the position and line-of-sight of the operator indicated by the position line-of-sight information output from the position line-of-sight information acquisition unit 11. If the position and line-of-sight direction of the operator at the time of setting the non-display area and the position and line-of-sight direction at the time of measurement by the position / posture measurement device 2 do not match, the position line-of-sight information acquisition unit 11 outputs The coordinate information output from the non-display area setting unit 12 is corrected according to the position line-of-sight information.
For example, the coordinate information output from the non-display area setting unit 12 is parallelized according to the difference between the position indicated by the position line-of-sight information output from the position / line-of-sight information acquisition unit 11 and the position of the operator when setting the non-display area. Non-display area setting by performing processing to move or rotating the coordinate information according to the difference between the gaze direction indicated by the position gaze information and the operator's gaze direction when setting the non-display area The coordinate information output from the unit 12 is corrected.

When the display image generation unit 13 generates a display image in the line-of-sight direction and the non-display region calculation unit 14 calculates the non-display region, the non-display region information search unit 15 calculates the image in the non-display region in the display image. It is determined whether or not there is an object that the operator needs to recognize (step ST4).
For example, if the display image data stored by the display image data storage unit 3 is CG three-dimensional data, the positions of N objects in the display image are specified with reference to the three-dimensional data ( N is an integer greater than or equal to 0), and if there is an object located within the non-display area among the N objects, the object is considered to be an object that the operator needs to recognize, and the non-display area It is determined that there is an object that the operator needs to recognize in the image inside.

For example, if the display image data stored in the display image data storage unit 3 is all-around image data centered on an operator photographed by a plurality of cameras, the latest all-around image data and all the frames before one frame are obtained. A moving object is detected by comparing the surrounding image data. If a moving object appears in the non-display area, the moving object is considered an object that the operator needs to recognize. It is determined that an object that the operator needs to recognize exists in the image in the display area.
FIG. 3 is an explanatory diagram showing an example in the case where an object that the operator needs to recognize exists in the non-display area.

The CG image generation unit 16 generates a CG image notifying that an object exists when it is determined by the non-display area information search unit 15 that an object that the operator needs to recognize exists. (Step ST5).
For example, as a CG image notifying that an object exists, a circle image centering on the position where the object exists is generated.
Here, FIG. 4 is an explanatory diagram showing an example in which an object that the operator needs to recognize is in the non-display area. In the example of FIG. 4, the object that the operator needs to recognize is a dotted line ○ It is present at the position of. However, for the convenience of explanation, the dotted line ◯ is shown in FIG. 4 but is not drawn as a display image.
FIG. 5 is an explanatory diagram showing an example of a circle image centered on the position where the object exists. In the example of FIG. 5, the circle image centered on the position of the dotted line ○ is other than the non-display area. It is drawn in the area.

Here, an example of generating a circle image centered on the position where the object exists is shown as a CG image notifying that the object exists, but notifying that the object exists The CG image is not limited to a circle image.
For example, as shown in FIG. 6, an image of a plurality of straight lines (two straight lines in the example of FIG. 6) that intersect at a position where an object in the non-display area exists may be generated. .
In the example of FIG. 6, a plurality of straight line images that intersect at a position where an object exists are drawn in a region other than the non-display region.

When the display image generation unit 13 generates a display image in the line-of-sight direction and the non-display area calculation unit 14 calculates the non-display area, the image composition unit 17 deletes the image in the non-display area from the display image in the line-of-sight direction. (Steps ST6 and ST8). Hereinafter, an image obtained by deleting the image in the non-display area from the display image in the line-of-sight direction is referred to as a “display image after deletion”.
When deleting an image in the non-display area, for example, processing such as filling the non-display area with black is performed.

When the CG image is generated by the CG image generation unit 16 (when an object that the operator needs to recognize exists in the image in the non-display area), the image composition unit 17 deletes the CG image. The post-display image and the CG image are synthesized (step ST7), and the synthesized image is displayed on the transmissive display device 1 as shown in FIG. 5 or 6 (step ST9).
On the other hand, when the CG image is not generated by the CG image generation unit 16 (when there is no object that the operator needs to recognize in the image in the non-display area), the display image after the deletion is displayed. The image is displayed on the transmissive display device 1 (step ST9).
Note that the image in the non-display area is not displayed on the transmissive display device 1 as described above, but since the transmissive display device 1 is a transmissive display device, an object in the non-display area can be seen through. There is. However, it cannot be confirmed in a situation such as being hidden behind another object.

  As is apparent from the above, according to the first embodiment, the position line-of-sight information acquisition unit 11 that acquires position line-of-sight information indicating the position and line-of-sight direction of the operator, and no image is displayed in the transmissive display device 1. The display image accumulated by the display image data accumulation unit 3 with the non-display region setting unit 12 for setting a non-display region and the operator's position indicated by the position gaze information acquired by the position / gaze information acquisition unit 11 as a viewpoint. The display image generation unit 13 that generates a display image in the line-of-sight direction indicated by the position line-of-sight information from the data, and the operator needs to recognize the image in the non-display area in the display image generated by the display image generation unit 13 The non-display area information search unit 15 that determines whether or not a certain object exists, and the non-display area information search unit 15 determine that there is an object that the operator needs to recognize. In this case, a CG image generation unit 16 that generates a CG image notifying that an object is present is provided, and the image composition unit 17 converts an image in the non-display area from the display image generated by the display image generation unit 13. If the CG image is not generated by the CG image generation unit 16, the display image in which the image in the non-display area is deleted is displayed on the transparent display device 1, and the CG image generation unit 16 generates the CG image. If so, the CG image is synthesized with the display image from which the image in the non-display area is deleted, and the synthesized image is displayed on the transmissive display device 1. Is present in the non-display area, it is possible to notify the operator to that effect.

  Further, according to the first embodiment, the CG image generation unit 16 generates a circle image centered on the position where the object exists as a CG image notifying that the object exists, Since the image composition unit 17 is configured to compose the circle image generated by the CG image generation unit 16 with the display image in a region other than the non-display region, the position of the object existing in the non-display region is determined. The effect which can be notified to an operator is produced.

  Further, according to the first embodiment, the CG image generation unit 16 generates a plurality of straight line images intersecting at the position where the object exists as a CG image notifying that the object exists. Since the image synthesis unit 17 is configured to synthesize a plurality of straight line images generated by the CG image generation unit 16 with the display image in an area other than the non-display area, an object existing in the non-display area There is an effect that the operator can be notified of the position.

  In the first embodiment, the non-display area calculation unit 14 performs three-dimensional information (position line-of-sight information acquired by the position line-of-sight information acquisition unit 11, coordinate information set by the non-display area setting unit 12). Is used to calculate the non-display area of the transmissive display device 1, but the operator places a marker in the area that the operator wants to hide, and attaches the camera to the transmissive display device 1 that is an HMD. Alternatively, the viewing direction of the non-display area may be obtained in terms of image processing.

Embodiment 2. FIG.
In the first embodiment, the CG image generation unit 16 uses a circle image centered on the position where the object exists (or the object exists) as the CG image notifying that the object exists. A plurality of straight line images intersecting at a certain position) is shown, but a CG image capable of grasping the size of the object may be generated.

FIG. 7 is an explanatory diagram showing an example of a CG image capable of grasping the size of an object.
In the example of FIG. 7, a rectangle having a width that matches the horizontal length (horizontal direction in the figure) of an object existing in the non-display area is displayed above and below the non-display area, and the vertical direction of the object A rectangle having a height that matches the length (vertical direction in the figure) is displayed on the left and right sides of the non-display area.
The four rectangles are arranged at positions corresponding to the positions of the objects existing in the non-display area. From the width of the upper and lower rectangles and the height of the left and right rectangles, The size of an object existing in

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  DESCRIPTION OF SYMBOLS 1 Transmission type display apparatus, 2 Position / attitude measurement apparatus, 3 Display image data storage part, 4 Image composition apparatus, 11 Position gaze information acquisition part (position gaze information acquisition means), 12 Non-display area setting part (Non-display area setting) Means), 13 display image generation section (display image generation means), 14 non-display area calculation section (CG image generation means), 15 non-display area information search section (CG image generation means), 16 CG image generation section (CG Image generation means), 17 image composition section (image composition means).

Claims (5)

Position line-of-sight information acquisition means for acquiring position line-of-sight information indicating the position and line-of-sight direction of the operator, non-display area setting means for setting a non-display area that does not display an image in the transmissive display device, and acquisition of the position line-of-sight information Display image generation means for generating a display image in the line-of-sight direction indicated by the position line-of-sight information from the stored display image data, with the position of the operator indicated by the position-line-of-sight information acquired by the means as a viewpoint, and the display image generation If there is an object that needs to be recognized by the operator in the image in the non-display area in the display image generated by the means, a CG image is generated to notify that the object exists. An image in the non-display area is deleted from the display image generated by the CG image generating means and the display image generating means, and the CG image is generated by the CG image generating means. If the CG image is generated by the CG image generation means, the display image in which the image in the non-display area is deleted is displayed on the transparent display device, and the image in the non-display area is deleted. An image synthesizing apparatus comprising: an image synthesizing unit that synthesizes the CG image in an area other than the non-display area in the display image and displays the synthesized image on the transmissive display apparatus. The CG image generating means, a CG image notifying that the object is present, to produce an image of a circle centered on the position where the object is present,
The image synthesizing unit synthesizes the image of the circle generated by the CG image generating unit with an area other than the non-display area in the display image from which the image in the non-display area is deleted. The image composition apparatus according to 1. The CG image generating means, a CG image notifying that the object is present, to produce an image of a plurality of straight lines that intersect at a position where the object is present,
The image synthesizing unit synthesizes the plurality of straight line images generated by the CG image generating unit into an area other than the non-display area in the display image from which the image in the non-display area is deleted. The image composition device according to claim 1. The CG image generating means, image synthesizing apparatus according to claim 1, wherein generating a CG image which can grasp the magnitude of the object. The position line-of-sight information acquisition means acquires position line-of-sight information acquisition processing step for acquiring position line-of-sight information indicating the position and line-of-sight direction of the operator, and the display image generation means includes a non-display area where no image is displayed in the transmissive display device. The non-display area setting processing step to be set, and the display image generating means, from the accumulated display image data, as the viewpoint, with the position of the operator indicated by the position visual axis information acquired in the positional visual axis information acquisition processing step as the viewpoint. A display image generation processing step for generating a display image in the line-of-sight direction indicated by the line-of-sight information, and a CG image generation means that operates in the image in the non-display area in the display image generated in the display image generation processing step. If there is an object that the person needs to recognize, a CG image generation processing step for generating a CG image notifying that the object exists, The generating means deletes the image in the non-display area from the display image generated in the display image generation processing step, and if no CG image is generated in the CG image generation processing step, If the display image from which the image has been deleted is displayed on the transparent display device and a CG image has been generated in the CG image generation processing step, the display image from which the image in the non-display area has been deleted is not the non-display area. An image synthesis method comprising: synthesizing the CG image in the area of the image, and displaying the synthesized image on the transmissive display device.

Images