JP2020088591A - Image processing system and image processing apparatus - Google Patents

Abstract

To provide an image processing system and an image processing apparatus capable of recording a first-person view point image of a worker, without making the worker wear a heavy electronic device such as a camera.SOLUTION: The video processing apparatus corrects the distortion of a curved mirror image reflected in goggles with a marker as a clue and generates image data in the direction of the goggles. The worker no longer needs to wear a heavy camera on his or her head during a work and can obtain image data of his or her view point only by wearing light-weight goggles.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video processing system and a video processing device.

2. Description of the Related Art Today, applications using virtual reality (hereinafter abbreviated as “VR”) technology have begun to spread widely for industrial applications, consumer applications, and the like. When an image is displayed on a goggle type head mounted display (hereinafter referred to as "HMD" (Head Mount Display)), the wearer of the HMD can feel as if he/she is in the world of the image displayed on the HMD. ..
Such a first-person viewpoint video recording the appearance from the self-viewpoint enables recording, transmission, and presentation of highly immersive video information. Therefore, attempts have been made to record various operations in VR, early learning of operations by allowing HMD wearers to relive those operations, and the use of such VR is expanding.

Patent Document 1 discloses the technical content of an information processing apparatus that easily realizes movement in a virtual space.
Patent Document 2 discloses the technical content of generating a three-dimensional image of an imaged object from camera images from multiple viewpoints.

JP, 2008-147498, A JP, 2017-130008, A

Surgery is a specific example of first-person perspective video for the purpose of learning work. Surgery, which requires advanced medical technology, has been difficult to master until now without the existence of patients, but by capturing a video of the valuable surgical procedure, it is possible to acquire the skill of surgery. It has become possible to speed up. Further, even if the moving image is not taken from a long distance with a general video camera, if the moving image can be taken from the viewpoint of the surgeon, it is expected that the skill of the surgeon can be confirmed more clearly.
However, the surgery performed by the surgeon is extremely delicate, unforgivable, and requires high concentration. For this reason, in the operation with an accessory such as a video camera attached, the weight of the video camera and the wiring of the image transmission cable interfere with the operation of the surgeon, and cause the surgeon's concentration to be impaired. It was difficult to do in reality.

The present invention solves such a problem, and an image processing system and an image capable of recording a spatial environment such as a first-person viewpoint image of an operator without attaching an accessory such as a heavy electronic device such as a camera to the operator. An object is to provide a processing device.

In order to solve the above-mentioned problems, an image processing system of the present invention includes a reflector that reflects an arbitrary object, a camera that can image an arbitrary object reflected by the reflector, and a position that can be directly imaged from the camera. The first marker that exists, the second marker that allows the camera to capture the mirror image reflected by the reflector, and the position of the first marker that exists in the video data output by the camera will identify the space and reflect from the video data. And an image processing device that cuts out an image portion of the body, grasps the distortion caused by the reflector from the shape of the second marker included in the image portion of the reflector, and corrects the image portion of the reflector to a flat state. .

According to the present invention, it is possible to generate a visual field image from an arbitrary spatial point without disposing an imaging device at an image observation point such as mounting a heavy electronic device on the operator, and thus a first-person perspective image of the worker. Etc. can be recorded.
Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a schematic diagram explaining the use state of the image processing system which is an example of an embodiment of the present invention. It is a block diagram which shows the hardware constitutions of a video processing apparatus. It is a block diagram which shows the software function of a video processing apparatus. It is a schematic diagram showing a relation of an omnidirectional camera, a marker, goggles, and a subject. It is a schematic diagram showing an embodiment of a passenger car carrying a vehicles auxiliary monitor device concerning a modification of the present invention. It is an example of the third embodiment of the present invention, and is a schematic diagram illustrating a usage state of the video processing system of the present invention. It is a block diagram which shows a part of software function of a video processing apparatus.

The video processing system according to the first embodiment of the present invention captures an image of goggles worn by an operator such as a surgeon with an omnidirectional camera or a wide-angle camera. The goggles have the characteristic of a convex mirror and reflect the subject at a wide angle. Then, the image processing system cuts out the goggles from the image of the camera, detects the direction of the goggles, and converts the goggles into an image viewed from the viewpoint of the goggles wearer.

[First Embodiment: Video Processing System 101: Outline]
FIG. 1 is an example of the first embodiment of the present invention, and is a schematic diagram illustrating a usage state of a video processing system 101 of the present invention. FIG. 1 shows a situation in which an omnidirectional camera 104 takes an image of a surgeon 102a and a nurse 102b performing a predetermined operation on a patient 103 as an object in an operating room. In addition, when the surgeon 102a and the nurse 102b are not distinguished, they are collectively referred to as an operator 102.

The image processing system 101 includes an omnidirectional camera 104, an image processing device 105, a marker 106, and goggles 107 having the features of a convex mirror.
The omnidirectional camera 104, which is also called an omnidirectional camera or the like, is a camera capable of photographing 360° in all directions. The photographed image is curved in a spherical shape, but it can be used as a planar image by correcting the curvature or distortion of the image with a predetermined video processing device. The omnidirectional camera 104 is a camera that is widely used mainly for applications such as surveillance cameras and security cameras.

The video processing device 105 is composed of a known information processing device such as a personal computer. The information processing device functions as the video processing device 105 by reading and executing a program for operating as the video processing device 105.
The marker 106 has large asymmetrical geometric patterns drawn vertically and horizontally. As described above, the use of the markers having asymmetrical patterns on the top, bottom, left, and right enables the computer to clearly determine the top, bottom, left, and right of the marker 106. The marker 106 is placed, for example, near the subject. As an example of the marker 106 shown in FIG. 1, operators for four arithmetic operations are drawn in a rectangular frame. The marker 106 may be printed on paper or drawn on a thin plate. Alternatively, it may be printed directly on the operating table 108 in the operating room as shown in FIG. The pattern of the marker 106 is not limited to that shown in FIG.
By grasping the relative positional relationship between the omnidirectional camera 104 and the marker 106, the image processing device 105 can set a reference for the virtual three-dimensional space formed inside the image processing device 105. The marker 106 is necessary for this purpose.

The goggles 107 are worn by a worker 102 such as a surgeon 102a and a nurse 102b who perform a predetermined work. The lens of the goggles 107 reflects the external scenery so that the omnidirectional camera 104 can take a picture. There are several possible methods for the omnidirectional camera 104 to capture the external scenery reflected by the lens of the goggles 107.
One is a method using a well-known magic mirror. The lens of the goggles 107 is, for example, a transparent synthetic resin plate such as acryl, to which a half mirror film made of a material such as a dielectric multilayer film or a metal thin film is attached. The surface of the lens of the goggles 107 has a curved shape similar to a spherical surface, and reflects the external scenery like a mirror. That is, the lens of the goggles 107 has a characteristic of a convex mirror. The goggles 107 having the features of such a convex mirror are on the market for winter sports such as skiing and snowboarding.
The other is that while using a simple mirror-finished plate for the lens of the goggles 107, the omnidirectional camera 104 uses a polarization camera, which is a camera that preferentially captures only the polarized component of the incident light. This is a method of photographing only the lens portion of 107.

Further, a goggle 107 having a lens surface made of a commonly-used transparent plastic having only a glossy surface and not subjected to mirror finishing (polycarbonate resin or acrylic resin is often used), The present invention can be applied even when an inexpensive commercially available camera 104 that is not a polarization camera or the like is used, as long as a desired image range in the operating room can be imaged. Even if the contrast of the reflected image obtained from the glossy surface of the lens of the goggles 107 is low and the ratio of the image signal and noise (S/N ratio) is low, by performing image processing such as HDR (High Dynamic Range) and dehaze. It is possible to implement the present invention by using the reflection image of the goggles 107 and the image of the commercially available camera 104.
It should be noted that a lens that has been subjected to antireflection processing, such as "non-glare processing" or "antiglare processing", cannot reflect an image in the first place, and thus is not included in the embodiments of the present invention.

The omnidirectional camera 104 images not only the marker 106 but also the goggles 107 worn by the worker 102. Then, the scenery viewed by the worker 102 is reflected on the goggles 107 and reflected on the omnidirectional camera 104. This scenery also includes the subject viewed by the worker 102. In FIG. 1, the patient 103 corresponds to the subject. Further, the marker 106 is also reflected in the goggles 107.

The image processing apparatus 105 cuts out the image reflected in the goggles 107 by using the marker 106 reflected in the goggles 107 as a clue, and corrects the distorted landscape appearing in the reflected images. Then, the image processing apparatus 105 calculates the coordinate information and the posture information of the goggles 107 in the virtual three-dimensional space internally formed, and based on the coordinate information and the posture information of the goggles 107, the worker 102 wearing the goggles 107 The image estimated to be viewed is cut out from the landscape that has undergone correction processing.

[Video processing device 105]
FIG. 2 is a block diagram showing the hardware configuration of the video processing device 105.
The video processing device 105 including a personal computer includes a CPU 202, a ROM 203, a RAM 204, a display unit 205, an operation unit 206, a nonvolatile storage 207 connected to the bus 201, and a serial interface 208 to which the omnidirectional camera 104 is connected. The serial interface 208 is, for example, a well-known USB.

FIG. 3 is a block diagram showing software functions of the video processing device 105. In FIG. 3, the information processing function block is shown by a solid line frame, and the data and information are shown by a one-dot chain line frame.
The omnidirectional captured image data 301 output by the omnidirectional camera 104 is input to the first marker detection processing unit 302.
The first marker detection processing unit 302 searches for the marker 106 included in the omnidirectional captured image data 301 based on the marker information 303 stored in the non-volatile storage 207. Then, the omnidirectional camera marker posture coordinate information 304, which is the position information and the posture information in the omnidirectional captured image data 301, is output. The marker information 303 includes information about the shape and size of the marker 106. The shape of the marker 106 may be image data of the marker 106.
The omnidirectional camera marker posture coordinate information 304 is input to the omnidirectional camera position estimation processing unit 305.
The omnidirectional camera position estimation processing unit 305 performs estimation calculation processing based on the omnidirectional camera marker posture coordinate information 304 and outputs omnidirectional camera coordinate information 306, which is coordinate information of the omnidirectional camera 104 in the virtual three-dimensional space. To do.

On the other hand, the omnidirectional shot image data 301 output by the omnidirectional camera 104 is also input to the goggle area extraction processing unit 307.
The goggle area extraction processing unit 307 searches for the goggles 107 included in the omnidirectional captured image data 301. Then, the goggle region extraction processing unit 307 cuts out only the region of the goggles 107 from the omnidirectional captured image data 301, the first goggle region image data 308a, the second goggle region image data 308b,... Is output.
There are several methods for finding the goggles 107 from the omnidirectional image data 301. As an example, there is a method in which a polarization camera is used as the omnidirectional camera 104 and a region where strong reflection based on polarization is strongly obtained from the omnidirectional captured image data 301 is located. A method of mounting a near-infrared LED on the periphery of the goggles 107 and blinking the near-infrared LEDs at a predetermined cycle to find the position of the goggles 107 is also conceivable. Further, it is also possible to find a portion having a characteristic of a convex mirror in the image data by an image recognition process using AI (artificial intelligence) which has been rapidly developed in recent years.

The first goggle region image data 308a, the second goggle region image data 308b,... The nth goggle region image data 308n output by the goggle region extraction processing unit 307 are input to the second marker detection processing unit 309.
The second marker detection processing unit 309, like the first marker detection processing unit 302, based on the marker information 303 stored in the non-volatile storage 207, the first goggle area video data 308a and the second goggle area video data. 308b,... Locates the markers 106 shown in each of the nth goggle area image data 308n. Then, the second marker detection processing unit 309 uses the first goggle region image data 308a, the second goggle region image data 308b,... The information 310a, the second goggle marker attitude coordinate information 310b,..., The nth goggle marker attitude coordinate information 310n are output.

The first goggle marker attitude coordinate information 310a, the second goggle marker attitude coordinate information 310b,... The nth goggle marker attitude coordinate information 310n are input to the goggle coordinate estimation processing unit 311.
The goggle coordinate estimation processing unit 311 includes the first goggle marker attitude coordinate information 310a, the second goggle marker attitude coordinate information 310b,... The nth goggle marker attitude coordinate information 310n, the omnidirectional camera marker attitude coordinate information 304, and the omnidirectional camera. An estimation calculation process based on the coordinate information 306 is performed. Then, the goggle coordinate estimation processing unit 311 includes the first goggle posture coordinate information 312a, which is information including the position, the posture, and the distortion of the marker 106 reflected in the goggles 107 in the virtual three-dimensional space of the omnidirectional camera 104. The second goggles posture coordinate information 312b,... Outputs the nth goggles posture coordinate information 312n.

The processing itself of the second marker detection processing unit 309 is almost the same as the processing of the first marker detection processing unit 302, but the second marker detection processing unit 309 uses the first goggle area video data 308a, the second goggle area video data 308b, The marker 106 detected from the nth goggle area image data 308n is inverted by the goggles 107, and geometric distortion occurs. Note that, hereinafter, the convex mirror, the plane mirror, and the concave mirror are collectively referred to as "reflecting mirror". Further, in the reflection image obtained from the reflection mirror, the distortion of the reflection image caused by the shape and/or the direction of the reflection mirror is defined as “geometrical distortion”. Geometric distortion includes distortion caused by the shape (curved surface) of the reflecting mirror and projective distortion caused by the direction of the reflecting mirror. "Geometric distortion" is a generic term that includes these two types of distortion. Define as.
On the other hand, the first goggle posture coordinate information 312a, the second goggles posture coordinate information 312b,... The nth goggles posture coordinate information 312n are the first goggles region image data 308a, the second goggles region image data 308b,. It includes not only the position and orientation of the marker 106 existing in the data 308n but also information about the geometric distortion of the marker 106. The geometric distortion of the marker 106 is based on the curved shape of the goggles 107. Therefore, the goggle image generation processing unit 313 corrects the geometric distortion of the first goggle region image data 308a, the second goggle region image data 308b,... The nth goggle region image data 308n using the information regarding the geometric distortion of the marker 106. Then, it is possible to convert into flat video data.

Therefore, the goggle image generation processing unit 313 includes the first goggle region image data 308a, the second goggle region image data 308b,... The nth goggle region image data 308n, the first goggle posture coordinate information 312a, and the second goggle posture coordinate information. The operator 102 wearing the goggles 107 receives the nth goggle posture coordinate information 312n, the omnidirectional camera marker posture coordinate information 304(A), and the omnidirectional camera coordinate information 306(B). It is estimated that the image is generated. Thus, the goggle image generation processing unit 313 outputs the first goggle viewpoint image data 314a, the second goggle viewpoint image data 314b,... The nth goggle viewpoint image data 314n.

Further, the first goggle viewpoint image data 314a, the second goggle viewpoint image data 314b,... The nth goggle viewpoint image data 314n are added to the three-dimensional shape estimation processing unit 315, and the three-dimensional shape estimation processing unit 315 virtualizes the subject. Three-dimensional video data 316 is generated. Note that the processing of the three-dimensional shape estimation processing unit 315 is not necessarily essential and is processing performed as needed.

Since the image data stream output from the omnidirectional camera 104, which is a monocular camera, is only two-dimensional image data, the video processing device 105 cannot detect the orientation, depth, etc. of a three-dimensional subject as it is.
However, the image processing apparatus 105 can acquire multi-view images by simultaneously obtaining viewpoint images of a plurality of workers. As a result, the video processing device 105 can perform three-dimensional image processing.
A method in which the video device 105 performs three-dimensional image processing is disclosed in Patent Document 2 by the present inventor.

[Camera, subject and goggles 107 in 3D space]
FIG. 4 is a schematic diagram showing the relationship among the omnidirectional camera 104, the marker 106, the goggles 107, and the subject 401.
In the video processing device 105, the omnidirectional camera 104 directly photographs the marker 106, so that the first marker detection processing unit 302 grasps the relative positional relationship between the omnidirectional camera 104 and the marker 106. Separately from this, the image processing apparatus 105 acquires image data in which the omnidirectional camera 104 images the mirror image M402 of the marker 106 and the mirror image M403 of the subject 401 through the lens 107a of the goggles 107.

The goggles coordinate estimation processing unit 311 calculates the orientation of the goggles 107.
The mirror image of the marker 106 has geometric distortion due to the curved surface of the goggles 107. The goggle coordinate estimation processing unit 311 captures this geometric distortion through the second marker detection processing unit 309 and uses it as a clue to correct the geometric distortion of the mirror image M403 of the subject 401.
The goggles image generation processing unit 313 is looking from the viewpoint of the worker 102 who wears the goggles 107, based on the information about the orientation of the goggles 107 and the geometric distortion derived from the curved surface of the goggles 107, which is output by the goggles coordinate estimation processing unit 311. It is estimated that the image is generated.

[Second embodiment: vehicle auxiliary monitor device]
FIG. 5 is a schematic view showing an embodiment of a passenger vehicle equipped with a vehicle auxiliary monitor device according to the second embodiment of the present invention.
A wide-angle camera 502 called a front view monitor is attached to the front grill of the passenger car 501. The video processing device 105 described above is connected to the wide-angle camera 502. A display unit 205 of the image processing device 105 is installed on the dashboard on the driver's seat side of the passenger car 501, and the display unit 205 compensates for the blind spot of the driver.
In FIG. 5, the passenger car 501 is about to leave the garage. On the other hand, the child 504 is riding on a bicycle and approaching the blind spot of the driver who separates the wall 503.
A road reflecting mirror 505, which is also called a curve mirror, is installed in front of the passenger car 501. As is well known, the road reflector 505 is a convex mirror.

On the road, a road marking 506 of "Tomarare" is printed. The road marking 506 is directly visible from the wide-angle camera 502 of the passenger car 501, and the mirror image reflected by the road reflecting mirror 505 is also visible. That is, the road marking 506 serves as the marker 106 in the above-described embodiment.
The image processing device 105 stores the data of the road marking 506 in advance in the nonvolatile storage 207 as the marker 106, and cuts out the portion of the road reflector 505 from the image of the wide-angle camera 502. Then, the area where the driver becomes a blind spot is displayed on the display unit 205 in a state in which the geometric distortion of the mirror image obtained from the road reflecting mirror 505 is corrected.

In the second embodiment described above, a wide angle camera 502 is used instead of the omnidirectional camera 104. However, whether the camera is capable of shooting in all directions or at a wide angle is not essential. Instead of the wide-angle camera 502, a plurality of cameras may be used to obtain a required angle of view.
Further, in the above-described modified example, the goggles 107 do not exist, but a convex mirror called a road reflecting mirror 505 exists.

[Third Embodiment: Video Processing System 601: Overall Configuration]
FIG. 6 is an example of the third embodiment of the present invention and is a schematic diagram illustrating a usage state of the video processing system 601 of the present invention.
The video processing system 601 shown in FIG. 6 is different from the video processing system 101 according to the first embodiment in FIG. 1 in that
<1> A first marker 602 for grasping the position of the omnidirectional camera 104 is attached to the wall of the operating room,
<2> The marker 106 in the first embodiment is that the mirror image reflected by the goggles 107 that is a reflector is present as the second marker 603 that can be photographed by the omnidirectional camera 104.

[Image processing device 604]
FIG. 7 is a block diagram showing a part of the software function of the video processing device 604.
The block diagram shown in FIG. 7 extracts only the first marker detection processing unit 302 and the second marker detection processing unit 309 of the video processing device 105 in FIG.
The first marker detection processing unit 302 reads the first marker information 701 indicating the characteristics of the first marker 602.
The first marker detection processing unit 302 searches for the first marker 602 included in the omnidirectional captured image data 301 based on the first marker information 701 stored in the non-volatile storage 207. Then, the omnidirectional camera marker posture coordinate information 304, which is the position information and the posture information in the omnidirectional captured image data 301, is output.

The second marker detection processing unit 309 reads the first marker information 702 indicating the characteristics of the second marker 603.
The second marker detection processing unit 309, like the first marker detection processing unit 302, based on the second marker information 702 stored in the non-volatile storage 207, the first goggle area video data 308a and the second goggle area. The second marker 603 shown in each of the image data 308b,..., The nth goggle area image data 308n is searched for. Then, the second marker detection processing unit 309 uses the first goggle region image data 308a, the second goggle region image data 308b,... The information 310a, the second goggle marker attitude coordinate information 310b,..., The nth goggle marker attitude coordinate information 310n are output.

As shown in FIGS. 6 and 7, in the image processing system 601, a first marker 602 for grasping the position of the omnidirectional camera 104 and a mirror image reflected by the goggles 107 as a reflector are captured by the omnidirectional camera 104. It has a possible second marker 603. The video processing system 101 according to the first embodiment can be considered as an integrated first marker 602 and second marker 603.

[Reflector and reflector]
As a matter common to the first embodiment, the second embodiment, and the third embodiment, description has been made so far on the assumption that an article such as a lens having a glossy surface of a reflecting mirror or goggles 107 has a convex surface shape. did. However, although the range of use is limited, even a plane mirror or a concave mirror having an extremely slight distortion can be used in the image processing system according to the present invention. The goggle image generation processing unit 313 corrects the geometric distortion included in the image data of the reflector area based on the marker information 303 or the second marker information 702.
Here, the term "reflector" includes not only an article having a mirror-like finish such as a reflecting mirror, but also an article having a glossy surface which is not a mirror-like finish and capable of obtaining a reflected image with a low contrast. It is defined as a superordinate word.

That is, the video processing system 601 according to the embodiment of the present invention is
A reflector that reflects any object,
A camera that can shoot any target reflected by the reflector,
A first marker 602 existing at a position where it can be directly photographed from a camera,
A second marker 603 capable of capturing a mirror image reflected by the reflector from a camera;
While grasping the space from the position of the first marker existing in the video data output by the camera, the video portion of the reflector is cut out from the video data, and the shape of the second marker included in the video portion of the reflector is used to reflect the shape of the second marker. An image processing device 604 for grasping the generated distortion and correcting the image portion of the reflector into a flat state.
One of the specific examples of the reflector is the goggles 107 in the first and third embodiments, and the other is the road reflector 505 in the second embodiment, that is, a convex mirror. Also, the reflector need not necessarily be a mirror finish. The specific examples of these reflectors are a list of examples, and other articles may be envisioned. For example, in the modification of FIG. 5, a reflecting plate made of acrylic or the like having a convex mirror shape may be installed in front of the passenger car 501 instead of the road reflecting mirror 505, and the wide-angle camera 502 may be a polarization camera.

When reviewing the video processing device 604 with the concept of a camera,
Omnidirectional video data 301 is video data,
The omnidirectional camera marker attitude coordinate information 304 is camera marker attitude coordinate information,
The omnidirectional camera position estimation processing unit 305 is a camera position estimation processing unit,
The omnidirectional camera coordinate information 306 is camera coordinate information.
When reviewing the image processing device 604 with the superordinate concept of a reflector,
The goggle area extraction processing unit 307 is a reflector area extraction processing unit,
The goggle coordinate estimation processing unit 311 is a reflector coordinate estimation processing unit,
The goggle image generation processing unit 313 is a reflector image generation processing unit.
Also, the first goggle region image data 308a, the second goggle region image data 308b,... The nth goggle region image data 308n are the first reflector region image data, the second reflector region image data,... The nth reflector region. It is video data.
The first goggle marker attitude coordinate information 310a, the second goggle marker attitude coordinate information 310b,... The nth goggle marker attitude coordinate information 310n are the first reflector marker attitude coordinate information, the second reflector marker attitude coordinate information,. It is reflector marker posture coordinate information.
The first goggle posture coordinate information 312a, the second goggle posture coordinate information 312b,... The nth goggle posture coordinate information 312n are the first reflector posture coordinate information, the second reflector posture coordinate information,... The nth reflector posture coordinate information. Is.
The first goggle viewpoint image data 314a, the second goggle viewpoint image data 314b,... The nth goggle viewpoint image data 314n are the first reflector viewpoint image data, the second reflector viewpoint image data,... The nth reflector viewpoint image data. Is.
The reflector image generation processing unit includes first reflector region image data, second reflector region image data,... Nth reflector region image data, first reflector posture coordinate information, second reflector posture coordinate information, ... receives the n-th reflector posture coordinate information, the camera marker posture coordinate information, and the camera coordinate information, and generates reflector viewpoint image data estimated to be the direction in which the reflector is facing.
Further, the reflector image generation processing unit, when the reflector region image data is image data including geometric distortion derived from the shape of the convex reflector, converts the reflector region image data generated by the shape of the convex reflector. By correcting the geometric distortion, the image portion of the convex reflector is corrected to a flat state.

In the embodiment of the present invention, the video processing system 101 is disclosed.
The image processing device 105 corrects the geometric distortion of the curved mirror image reflected on the goggles 107 by using the marker 106 as a clue to generate image data in the direction in which the goggles 107 face. The worker 102 does not need to wear a heavy camera on his head during the work, and only by wearing the lightweight goggles 107, the image data of the viewpoint of the worker 102 can be obtained.
The image processing system 101 can be expected to have a great effect on the acquisition of a surgical operation, which is less likely to be acquired. Further, not only the surgical operation but also various work images can be easily obtained from the viewpoint of the operator without attaching a camera to the head of the operator 102.
Furthermore, if a plurality of goggles 107 and convex mirrors are present around the subject, three-dimensional stereoscopic data of the subject can be easily obtained.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and other modifications and application examples are possible without departing from the gist of the present invention described in the claims. including.

101... Image processing system, 102... Worker, 102a... Surgeon, 102b... Nurse, 103... Patient, 104... Omnidirectional camera, 105... Image processing device, 106... Marker, 107... Goggles, 107a... Lens, 108 ...Operating table, 201... Bus, 202... CPU, 203... ROM, 204... RAM, 205... Display section, 206... Operation section, 207... Nonvolatile storage, 208... Serial interface, 301... Omnidirectional image data, 302 ... first marker detection processing unit, 303... marker information, 304... omnidirectional camera marker posture coordinate information, 305... omnidirectional camera position estimation processing unit, 306... omnidirectional camera coordinate information, 307... goggle area extraction processing unit, 308a ... 1st goggle area image data, 308b... 2nd goggle area image data, 308n... nth goggle area image data, 309... 2nd marker detection processing part, 310a... 1st goggle marker posture coordinate information, 310b... 2nd goggles Marker posture coordinate information, 310n... Nth goggles posture coordinate information, 311... Goggles coordinate estimation processing unit, 312a... First goggles posture coordinates information, 312b... Second goggles posture coordinates information, 312n... Nth goggles posture coordinates information, 313... Goggle image generation processing unit, 314a... First goggle viewpoint image data, 314b... Second goggle viewpoint image data, 314n... Nth goggle viewpoint image data, 315... Three-dimensional shape estimation processing unit, 316... Virtual three-dimensional image Data, 401... Subject, 501... Passenger car, 502... Wide angle camera, 503... Wall, 504... Child, 505... Road reflector, 506... Road marking, 601... Image processing system, 602... First marker, 603... No. Two markers, 604... Image processing device, 701... First marker information, 702... Second marker information

Claims (9)

A reflector that reflects any object,
A camera capable of photographing any object reflected by the reflector,
A first marker existing at a position where it can be directly photographed from the camera,
A second marker that allows the mirror image reflected by the reflector to be photographed from the camera,
While grasping the space from the position of the first marker existing in the video data output by the camera, cutting out the video portion of the reflector from the video data, the second marker of the second marker included in the video portion of the reflector. An image processing system, comprising: an image processing device that grasps a distortion caused by the reflector from a shape and corrects an image portion of the reflector to a planar state. The reflector is a convex reflector,
The image processing device, in the mirror image obtained from the convex reflector, corrects the geometric distortion of the mirror image caused by the shape of the convex reflector to make the image portion of the convex reflector in a planar state. The image processing system according to claim 1, wherein correction is performed. The video processing device,
A first marker detection processing unit that searches for the first marker appearing in the video data and outputs camera marker attitude coordinate information;
A camera position estimation processing unit that performs estimation calculation processing based on the camera marker posture coordinate information and outputs camera coordinate information that is coordinate information in the virtual three-dimensional space of the camera;
A reflector region extraction processing unit that outputs reflector region image data obtained by cutting out one or more reflectors from the image data;
Second, searching for a mirror image of the second marker reflected in one or more of the reflector area image data, and outputting reflector marker attitude coordinate information that is position information and attitude information in the one or more reflector area image data. A marker detection processing unit,
The second image of the one or more reflectors in the virtual three-dimensional space of the camera based on the one or more reflector marker posture coordinate information, the camera marker posture coordinate information, and the camera coordinate information. A reflector coordinate estimation processing unit that outputs reflector posture coordinate information that is information including the position, posture, and distortion of the marker,
Reflection estimated to be the direction in which the reflector is facing based on one or more of the reflector region image data, one or more of the reflector posture coordinate information, the camera marker posture coordinate information, and the camera coordinate information. In addition to generating body viewpoint image data, when the reflector region image data is image data of the convex reflector, corrects geometric distortion of the reflector region image data caused by the shape of the convex reflector. The image processing system according to claim 2, further comprising: a reflector image generation processing unit that corrects the image portion of the convex reflector into a planar state.
The image processing device further includes
The image processing system according to claim 3, further comprising: a three-dimensional shape estimation processing unit that generates virtual three-dimensional image data of the subject based on the plurality of reflector viewpoint image data. The image processing system according to claim 2, 3 or 4, wherein the reflector is goggles. The image processing system according to claim 2, 3 or 4, wherein the reflector is a reflecting mirror. The image processing system according to claim 5, wherein the first marker and the second marker are the same. A reflector that reflects an arbitrary object, a camera that can shoot an arbitrary object reflected by the reflector, and a mirror image that is present at a position that allows direct shooting from the camera and that is also reflected by the reflector. The first marker that can be photographed from the first marker, the second marker that the mirror image reflected by the reflector can be photographed from the camera, and the position of the first marker that is present in the video data output by the camera, to grasp the space In addition, the image portion of the reflector is cut out from the image data, the distortion caused by the reflector is grasped from the shape of the second marker included in the image portion of the reflector, and the image portion of the reflector is grasped. Is a video processing device in a video processing system comprising:
The video processing device,
A first marker detection processing unit that searches for the first marker appearing in the video data and outputs camera marker attitude coordinate information;
A camera position estimation processing unit that performs estimation calculation processing based on the camera marker posture coordinate information and outputs camera coordinate information that is coordinate information in the virtual three-dimensional space of the camera;
A reflector region extraction processing unit that outputs reflector region image data obtained by cutting out one or more reflectors from the image data;
Second, searching for a mirror image of the second marker reflected in one or more of the reflector area image data, and outputting reflector marker attitude coordinate information that is position information and attitude information in the one or more reflector area image data. A marker detection processing unit,
The second image of the one or more reflectors in the virtual three-dimensional space of the camera based on the one or more reflector marker posture coordinate information, the camera marker posture coordinate information, and the camera coordinate information. A reflector coordinate estimation processing unit that outputs reflector posture coordinate information that is information including the position, posture, and distortion of the marker,
Reflection estimated to be the direction in which the reflector is facing based on one or more of the reflector region image data, one or more of the reflector posture coordinate information, the camera marker posture coordinate information, and the camera coordinate information. Compensating geometric distortion of the reflector region image data generated by the shape of the convex reflector when the body region image data is generated and the reflector region image data is image data by a convex reflector. An image processing apparatus comprising: a reflector image generation processing unit that corrects an image portion of the convex reflector into a flat state. Furthermore,
The image processing apparatus according to claim 8, further comprising: a three-dimensional shape estimation processing unit that generates virtual three-dimensional image data of a subject based on a plurality of the reflector viewpoint image data.

Images