JP7181589B2 - Video processing system and video processing device - Google Patents

Description

The present invention relates to a video processing system and a video processing device.

Today, applications using virtual reality (hereinafter abbreviated as “VR”) technology are beginning to spread widely for industrial use, consumer use, and the like. When an image is projected on a goggle-type head-mounted display (hereinafter abbreviated as "HMD" (Head Mount Display)), the wearer of the HMD can feel as if he/she is in the world of the image projected on the HMD. .
Such a first-person viewpoint video, which records the way of viewing from one's own viewpoint, enables recording, transmission, and presentation of video information with a high sense of immersion. Therefore, attempts have been made to use VR to record various types of work, and to allow the HMD wearer to experience the work at an early stage, and the use of such VR is becoming more widespread.

Patent Literature 1 discloses the technical content of an information processing apparatus that facilitates movement in a virtual space.
Japanese Patent Laid-Open No. 2002-200002 discloses a technique for generating a three-dimensional image of an object to be imaged from camera images taken from multiple viewpoints.

JP 2018-147498 A Japanese Patent Application Laid-Open No. 2017-130008

Surgery is a specific example of first-person-view video for the purpose of learning operations. Surgery requires advanced medical technology, and until now it was difficult to master without the presence of the patient. It is now possible to move forward. In addition, if the moving image can be captured from the perspective of the surgeon instead of using a general video camera to capture the patient from a distance, it can be expected that the surgeon's skill can be confirmed more clearly.
However, the surgeon's treatment is extremely delicate and failure is not allowed, requiring a high degree of concentration. For this reason, the weight of the video camera and the wiring of the video transmission cable hinder the operation of the surgeon, hindering the concentration of the surgeon. It was difficult to do in reality.

The present invention solves this problem, and a video processing system and video that can record a spatial environment such as a first-person viewpoint video of a worker without having the worker wear an accessory such as a heavy electronic device such as a camera. It is an object of the present invention to provide a processing apparatus.

In order to solve the above problems, the image processing system of the present invention includes a reflector that reflects an arbitrary object, a camera capable of photographing the arbitrary object reflected by the reflector, and a The space is grasped from the position of the existing first marker, the second marker whose mirror image reflected by the reflector can be captured by the camera, and the position of the first marker existing in the video data output by the camera, and the space is reflected from the video data. and an image processing device that cuts out an image portion of the body, grasps distortion caused by the reflector from the shape of a second marker included in the image portion of the reflector, and corrects the image portion of the reflector to a flat state. .

According to the present invention, it is possible to generate a field-of-view image from an arbitrary spatial point without arranging an imaging device at the image observation point, such as having the worker wear a heavy electronic device, and the operator's first-person-view image. etc., can be recorded.
Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a schematic diagram illustrating a usage state of a video processing system, which is an example of an embodiment of the present invention; FIG. It is a block diagram which shows the hardware constitutions of a video processing apparatus. 3 is a block diagram showing software functions of the video processing device; FIG. 1 is a schematic diagram showing the relationship among an omnidirectional camera, markers, goggles, and a subject; FIG. It is a schematic diagram showing an embodiment of a passenger car which carries a vehicle auxiliary monitor device concerning a modification of the present invention. It is an example of a third embodiment of the present invention, and is a schematic diagram for explaining the usage state of the video processing system of the present invention. 3 is a block diagram showing part of the software functions of the video processing device; FIG.

A video processing system according to the first embodiment of the present invention captures an image reflected in goggles worn by an operator such as a surgeon with an omnidirectional camera or a wide-angle camera. Goggles have the characteristics of a convex mirror and reflect the subject at a wide angle. Then, the image processing system cuts out the goggles portion from the image of the camera, detects the orientation of the goggles, and converts it into an image viewed from the viewpoint of the goggles wearer.

[First Embodiment: Video Processing System 101: Overview]
FIG. 1 is an example of the first embodiment of the present invention, and is a schematic diagram for explaining the usage state of a video processing system 101 of the present invention. FIG. 1 shows how a surgeon 102a and a nurse 102b perform a predetermined treatment on a patient 103 as an object in an operating room, photographed by an omnidirectional camera 104. FIG. The surgeon 102a and the nurse 102b are collectively referred to as the operator 102 when not distinguished from each other.

The image processing system 101 comprises an omnidirectional camera 104, an image processing device 105, a marker 106, and goggles 107 having convex mirror features.
The omnidirectional camera 104 is also called an omnidirectional camera or the like, and is a camera capable of photographing in all directions of 360°. The captured image is curved into a spherical shape, but it can be used as a planar image by correcting the curvature and distortion of the image with a predetermined video processing device. The omnidirectional camera 104 is a camera widely used mainly for surveillance cameras, security cameras, and the like.

The video processing device 105 is composed of an information processing device such as a well-known personal computer. The information processing device functions as the video processing device 105 by reading and executing a program for operating as the video processing device 105 .
The marker 106 has a large, vertically asymmetrical geometric pattern drawn on it. By using a marker having a vertically and horizontally asymmetrical pattern in this manner, the computer can clearly distinguish the top, bottom, left, and right of the marker 106 . This marker 106 is placed, for example, near the subject. As an example of the marker 106 shown in FIG. 1, operators of four arithmetic operations are drawn in a rectangular frame. The marker 106 may be printed on paper or drawn on a thin board. It may also be printed directly on the operating table 108 in the operating room as shown in FIG. Note that the pattern of the marker 106 is not limited to that shown in FIG.
By grasping the relative positional relationship between the omnidirectional camera 104 and the marker 106, the video processing device 105 can provide a reference for the virtual three-dimensional space that the video processing device 105 internally forms. Markers 106 are necessary for this purpose.

The goggles 107 are worn by operators 102, such as a surgeon 102a and a nurse 102b, who perform predetermined tasks. The lens of the goggles 107 reflects the outside scenery so that the omnidirectional camera 104 can photograph it. Several methods are conceivable as a mechanism for the omnidirectional camera 104 to capture the external scenery reflected by the lenses of the goggles 107 .
One is a method using a well-known magic mirror. The lens of the goggles 107 is, for example, a plate made of transparent synthetic resin such as acrylic, to which a half-mirror film made of a material such as a dielectric multilayer film or a metal thin film is attached. The surface of the lens of the goggles 107 has a curved shape resembling a spherical surface and reflects the scenery of the outside world like a mirror. That is, the lens of goggles 107 has the characteristics of a convex mirror. The goggles 107 having such convex mirror features are on the market for use in winter sports such as skiing and snowboarding.
The other is to use a simple mirror-finished plate for the lens of the goggles 107, and use a polarization camera as the omnidirectional camera 104, which preferentially captures only the polarized component of the incident light, thereby making the goggles In this method, only the lens portion of 107 is photographed.

Further, goggles 107 having lens surfaces made of transparent plastic (polycarbonate resin and acrylic resin are often used) that have only a glossy surface and are not mirror-finished, which are commonly used; Even if an inexpensive commercially available camera 104 other than a polarizing camera or the like is used, the present invention can be applied if the target image range in the operating room can be imaged. Even if the contrast of the reflected image obtained from the glossy surface of the lens of the goggles 107 is low and the ratio of the video signal to noise (S/N ratio) is low, image processing such as HDR (High Dynamic Range) and dehaze can be applied. , the reflected image of the goggles 107, and the image of the commercially available camera 104 can be used to implement the present invention.
It should be noted that a lens subjected to anti-reflection processing, which is called “non-glare processing” or “anti-glare processing”, cannot reflect an image in the first place, and therefore is not the object of the embodiments of the present invention.

The omnidirectional camera 104 captures not only the marker 106 but also the goggles 107 worn by the worker 102 . Then, the scenery seen by the worker 102 is reflected on the goggles 107 and captured by the omnidirectional camera 104 . This scenery also includes a subject that the worker 102 sees. In FIG. 1, the patient 103 corresponds to the subject. Furthermore, the marker 106 is also reflected on the goggles 107 .

Using the marker 106 reflected in the goggles 107 as a clue, the image processing device 105 cuts out the image reflected in the goggles 107 and performs correction processing on the distorted scenery appearing in the reflected image. Then, the coordinate information and posture information of the goggles 107 in the virtual three-dimensional space internally formed by the image processing device 105 are calculated, and based on the coordinate information and posture information of the goggles 107, the operator 102 wearing the goggles 107 is The image that is presumed to be seen is cut out from the scenery that has undergone correction processing.

[Video processing device 105]
FIG. 2 is a block diagram showing the hardware configuration of the video processing device 105. As shown in FIG.
A video processing apparatus 105 comprising a personal computer includes a CPU 202, a ROM 203, a RAM 204, a display unit 205, an operation unit 206, a nonvolatile storage 207, and a serial interface 208 to which the omnidirectional camera 104 is connected. The serial interface 208 is, for example, the well-known USB.

FIG. 3 is a block diagram showing software functions of the video processing device 105. As shown in FIG. In FIG. 3, information processing function blocks are indicated by solid-line frames, and data and information are indicated by dashed-dotted line frames.
Omnidirectional image data 301 output by the omnidirectional camera 104 is input to the first marker detection processing unit 302 .
The first marker detection processing unit 302 finds the marker 106 appearing in the omnidirectional image data 301 based on the marker information 303 stored in the nonvolatile storage 207 . Then, omnidirectional camera marker orientation coordinate information 304, which is the position information and orientation information in the omnidirectional image data 301, is output. The marker information 303 consists of information regarding the shape and size of the marker 106 . The shape of the marker 106 may be image data of the marker 106 .
The omnidirectional camera marker orientation coordinate information 304 is input to the omnidirectional camera position estimation processing unit 305 .
The omnidirectional camera position estimation processing unit 305 performs estimation calculation processing based on the omnidirectional camera marker orientation coordinate information 304 and outputs omnidirectional camera coordinate information 306, which is coordinate information in the virtual three-dimensional space of the omnidirectional camera 104. do.

On the other hand, the omnidirectional image data 301 output by the omnidirectional camera 104 is also input to the goggle area extraction processing unit 307 .
The goggles area extraction processing unit 307 finds the goggles 107 appearing in the omnidirectional image data 301 . Then, the goggle area extraction processing unit 307 extracts only the goggle area image data 308a, second goggle area image data 308b, . . . to output
There are several methods for finding the goggles 107 from the omnidirectional image data 301 . As an example, there is a method of using a polarization camera as the omnidirectional camera 104 and searching for an area where strong reflection based on polarized light is obtained from the omnidirectional image data 301 . A method of finding the position of the goggles 107 by attaching near-infrared LEDs to the periphery of the goggles 107 and blinking the near-infrared LEDs at a predetermined cycle is also conceivable. Furthermore, image recognition processing using AI (artificial intelligence), which has been rapidly developing in recent years, may be used to search for a portion having features of a convex mirror in image data.

The first goggle area image data 308a, the second goggle area image data 308b, .
Similar to the first marker detection processing unit 302, the second marker detection processing unit 309 generates first goggles area image data 308a and second goggles area image data based on the marker information 303 stored in the nonvolatile storage 207. 308b, . Then, the second marker detection processing unit 309 detects first goggle marker posture coordinates, which are position information and posture information in the first goggle area image data 308a, the second goggle area image data 308b, . Information 310a, second goggle marker posture coordinate information 310b, . . . nth goggle marker posture coordinate information 310n are output.

The first goggle marker posture coordinate information 310a, the second goggle marker posture coordinate information 310b, . . .
The goggle coordinate estimation processing unit 311 stores first goggle marker posture coordinate information 310a, second goggle marker posture coordinate information 310b, . An estimation calculation process based on the coordinate information 306 is performed. Then, the goggle coordinate estimation processing unit 311 generates first goggle posture coordinate information 312a, which is information including the position, posture, and distortion of the marker 106 captured by the goggles 107 in the virtual three-dimensional space of the omnidirectional camera 104; Second goggle posture coordinate information 312b, . . . n-th goggle posture coordinate information 312n are output.

The processing itself of the second marker detection processing unit 309 is almost the same as that of the first marker detection processing unit 302, but the second marker detection processing unit 309 generates first goggle area image data 308a, second goggle area image data 308b, . . The marker 106 detected from the n-th goggle area image data 308n is inverted by the goggles 107 and geometrically distorted. Hereinafter, convex mirrors, plane mirrors, and concave mirrors will be collectively referred to as "reflecting mirrors." Furthermore, in the reflected image obtained from the reflecting mirror, the distortion of the reflected image caused by the shape and/or direction of the reflecting mirror is defined as "geometric distortion". Geometric distortion includes distortion derived from the shape (curved surface) of the reflector and projection distortion derived from the direction of the reflector. defined as
On the other hand, the first goggle posture coordinate information 312a, the second goggle posture coordinate information 312b, . It includes not only the position and orientation of marker 106 present in data 308n, but also information about geometric distortion of marker 106. FIG. The geometric distortion of marker 106 is based on the curved shape of goggles 107 . Therefore, the goggle image generation processing unit 313 corrects the geometric distortion of the first goggle area image data 308a, the second goggle area image data 308b, . . . can be converted into planar video data.

Therefore, the goggle image generation processing unit 313 generates first goggle area image data 308a, second goggle area image data 308b, . 312b, . . . n-th goggle posture coordinate information 312n, omnidirectional camera marker posture coordinate information 304(A), and omnidirectional camera coordinate information 306(B) are received, and the operator 102 wearing the goggles 107 sees them. Generates an image that is estimated to be Thus, the goggle image generation processing unit 313 outputs first goggle viewpoint image data 314a, second goggle viewpoint image data 314b, . . . nth goggle viewpoint image data 314n.

Further, the first goggle viewpoint video data 314a, the second goggle viewpoint video data 314b, . 3D video data 316 is generated. It should be noted that the processing of the three-dimensional shape estimation processing unit 315 is not necessarily essential, and is processing that is performed as necessary.

Since the image data stream output by the omnidirectional camera 104, which is a monocular camera, is strictly two-dimensional image data, the video processing device 105 cannot detect the orientation, depth, etc. of a three-dimensional object.
However, the image processing device 105 can acquire multi-viewpoint images by simultaneously acquiring viewpoint images of a plurality of workers. As a result, the video processing device 105 can perform three-dimensional image processing.
A method for performing three-dimensional image processing by the video device 105 is disclosed in Patent Document 2 by the inventor of the present patent.

[Camera, shooting target, and goggles 107 in three-dimensional space]
FIG. 4 is a schematic diagram showing the relationship among the omnidirectional camera 104, the marker 106, the goggles 107, and the subject 401. As shown in FIG.
In the video processing device 105 , the omnidirectional camera 104 directly captures the marker 106 so that the first marker detection processing unit 302 grasps the relative positional relationship between the omnidirectional camera 104 and the marker 106 . Separately from this, the video processing device 105 acquires video data of the mirror image M402 of the marker 106 and the mirror image M403 of the subject 401 captured by the omnidirectional camera 104 through the lens 107a of the goggles 107. FIG.

The goggle coordinate estimation processing unit 311 calculates the orientation of the goggles 107 .
The mirror image of the marker 106 is geometrically distorted by the curved surface of the goggles 107 . The goggle coordinate estimation processing unit 311 captures this geometric distortion through the second marker detection processing unit 309 and uses it as a clue for correcting the geometric distortion of the mirror image M403 of the subject 401 .
The goggle image generation processing unit 313 sees from the viewpoint of the worker 102 wearing the goggles 107 based on the information about the orientation of the goggles 107 and the geometric distortion derived from the curved surface of the goggles 107 output by the goggle coordinate estimation processing unit 311. Generates an image that is estimated to be

[Second Embodiment: Vehicle Auxiliary Monitor Device]
FIG. 5 is a schematic diagram showing an embodiment of a passenger car equipped with a vehicle auxiliary monitor device according to a second embodiment of the present invention.
A wide-angle camera 502 called a front view monitor is attached to the front grill of a passenger car 501 . The wide-angle camera 502 is connected to the video processing device 105 described above. The display unit 205 of the video processing device 105 is installed on the dashboard on the driver's seat side of the passenger car 501, and the display unit 205 compensates for the driver's blind spot.
In FIG. 5, a passenger car 501 is leaving the garage. On the other hand, a child 504 riding a bicycle is approaching the driver's blind spot across the wall 503 .
A road reflector 505 also called a curve mirror is installed in front of the passenger car 501 . As is well known, road reflector 505 is a convex mirror.

A road marking 506 of "Stop" is printed on the road. This pavement marking 506 is directly visible from the wide-angle camera 502 of the passenger vehicle 501 and is also visible as a mirror image reflected through the road reflector 505 . That is, the road marking 506 plays the role of the marker 106 in the above-described embodiment.
The video processing device 105 stores the data of the road marking 506 as the marker 106 in the non-volatile storage 207 in advance, and extracts the road reflector 505 portion from the video of the wide-angle camera 502 . Then, the area that is the blind spot for the driver is displayed on the display unit 205 in a state in which the geometric distortion of the mirror image obtained from the road reflecting mirror 505 is corrected.

A wide angle camera 502 is used in place of the omnidirectional camera 104 in the second embodiment described above. However, it is not always essential whether the camera is capable of photographing in all directions or has a wide angle. Instead of the wide-angle camera 502, a plurality of cameras may be used to obtain the required angle of view.
In addition, instead of the goggles 107 being present in the modified example described above, a convex mirror called a road reflecting mirror 505 is present.

[Third Embodiment: Video Processing System 601: Overall Configuration]
FIG. 6 is an example of a third embodiment of the present invention, and is a schematic diagram for explaining the state of use of a video processing system 601 of the present invention.
The difference between the video processing system 601 shown in FIG. 6 and the video processing system 101 according to the first embodiment in FIG.
<1> A first marker 602 for grasping the position of the omnidirectional camera 104 is attached to the wall of the operating room;
<2> The marker 106 in the first embodiment exists as a second marker 603 whose mirror image reflected by the goggles 107 that is a reflector can be captured by the omnidirectional camera 104 .

[Video processing device 604]
FIG. 7 is a block diagram showing some of the software functions of the video processing device 604. As shown in FIG.
The block diagram shown in FIG. 7 extracts only the first marker detection processing unit 302 and the second marker detection processing unit 309 of the video processing device 105 in FIG.
First marker information 701 indicating the characteristics of the first marker 602 is read into the first marker detection processing unit 302 .
The first marker detection processing unit 302 finds the first marker 602 appearing in the omnidirectional image data 301 based on the first marker information 701 stored in the nonvolatile storage 207 . Then, omnidirectional camera marker orientation coordinate information 304, which is the position information and orientation information in the omnidirectional image data 301, is output.

First marker information 702 indicating the characteristics of the second marker 603 is read into the second marker detection processing unit 309 .
Similar to the first marker detection processing unit 302, the second marker detection processing unit 309 generates the first goggle area image data 308a, the second goggle area image data 308a, and the second goggle area image data 308a based on the second marker information 702 stored in the nonvolatile storage 207. The second marker 603 appearing in each of the image data 308b, . . . n-th goggle area image data 308n is found. Then, the second marker detection processing unit 309 detects first goggle marker posture coordinates, which are position information and posture information in the first goggle area image data 308a, the second goggle area image data 308b, . Information 310a, second goggle marker posture coordinate information 310b, . . . nth goggle marker posture coordinate information 310n are output.

As shown in FIGS. 6 and 7, the video processing system 601 captures a mirror image reflected by a first marker 602 for grasping the position of the omnidirectional camera 104 and the goggles 107, which is a reflector, from the omnidirectional camera 104. It has a second marker 603 that is possible. The image processing system 101 in the first embodiment can be considered as an integration of the first marker 602 and the second marker 603 .

[Reflector and Reflector]
As a matter common to the first, second, and third embodiments, the description so far has been made on the premise that an article such as a reflecting mirror or a lens having a glossy surface of goggles 107 has a convex shape. did. However, even a flat mirror or a concave mirror with a very slight distortion can be used in the image processing system according to the present invention, although the scope of use is limited. The goggle image generation processing unit 313 corrects the geometric distortion included in the image data of the reflector area based on the marker information 303 or the second marker information 702 .
Here, the reflector includes not only mirror-finished articles such as reflecting mirrors, but also non-mirror-finished articles that have a glossy surface and are capable of obtaining a low-contrast reflected image. It is defined as a word of a higher concept.

That is, the video processing system 601 according to the embodiment of the present invention is
a reflector that reflects any target;
a camera capable of photographing any object reflected by the reflector;
A first marker 602 that exists in a position that can be directly photographed from the camera;
a second marker 603 whose mirror image reflected by the reflector can be captured by the camera;
The space is grasped from the position of the first marker present in the image data output by the camera, the image portion of the reflector is cut out from the image data, and the reflector is used from the shape of the second marker included in the image portion of the reflector. and an image processing device 604 that recognizes the distortion that has occurred and corrects the image portion of the reflector to a flat state.
One specific example of the reflector is the goggles 107 in the first and third embodiments, and the other is the road reflector 505 in the second embodiment, that is, a convex mirror. Also, the reflector does not necessarily have to be mirror-finished. These specific examples of reflectors are exemplary listings and other articles can be envisioned. For example, in the modification of FIG. 5, instead of the road reflecting mirror 505, a reflecting plate such as acrylic having a convex mirror shape may be installed in front of the passenger car 501, and the wide-angle camera 502 may be composed of a polarizing camera.

Looking at the video processing device 604 from the higher-level concept of a camera,
The omnidirectional image data 301 is image data,
The omnidirectional camera marker orientation coordinate information 304 is camera marker orientation coordinate information,
The omnidirectional camera position estimation processing unit 305 is a camera position estimation processing unit,
The omnidirectional camera coordinate information 306 is camera coordinate information.
When the video processing device 604 is reviewed from the higher concept of reflector,
The goggle area extraction processing unit 307 is a reflector area extraction processing unit,
The goggle coordinate estimation processing unit 311 is a reflector coordinate estimation processing unit,
A goggle image generation processing unit 313 is a reflector image generation processing unit.
Also, the first goggle area image data 308a, the second goggle area image data 308b, . video data.
The first goggle marker posture coordinate information 310a, the second goggle marker posture coordinate information 310b, . This is reflector marker attitude coordinate information.
The first goggle posture coordinate information 312a, the second goggle posture coordinate information 312b, . is.
The first goggle-view image data 314a, the second goggle-view image data 314b, . is.
The reflector image generation processing unit generates first reflector area image data, second reflector area image data, . . . receives the n-th reflector orientation coordinate information, the camera marker orientation coordinate information, and the camera coordinate information, and generates reflector viewpoint image data that is estimated as the direction in which the reflector faces.
In addition, the reflector image generation processing unit generates reflector area image data generated by the shape of the convex reflector when the reflector area image data is image data including geometric distortion derived from the shape of the convex reflector. By correcting the geometric distortion, the image portion of the convex reflector is corrected to a flat state.

In embodiments of the present invention, a video processing system 101 is disclosed.
The image processing device 105 corrects the geometric distortion of the curved mirror image reflected on the goggles 107 using the markers 106 as clues, and generates image data in the direction in which the goggles 107 face. The worker 102 does not need to wear a heavy camera on the head during work, and can obtain video data of the worker 102's viewpoint only by wearing the light goggles 107 .
The video processing system 101 can be expected to be particularly effective in learning surgical operations, which have few opportunities to learn. In addition, it is possible to easily obtain video data from the viewpoint of the operator without attaching a camera to the head of the operator 102, not limited to surgical operations.
Furthermore, if a plurality of goggles 107 and convex mirrors exist around the subject, three-dimensional stereoscopic data of the subject can be obtained easily.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and other modifications and applications can be made without departing from the gist of the present invention described in the claims. including.

DESCRIPTION OF SYMBOLS 101... Image processing system, 102... Operator, 102a... Surgeon, 102b... Nurse, 103... Patient, 104... Omnidirectional camera, 105... Image processing apparatus, 106... Marker, 107... Goggles, 107a... Lens, 108 Operating table 201 Bus 202 CPU 203 ROM 204 RAM 205 Display unit 206 Operation unit 207 Non-volatile storage 208 Serial interface 301 Omnidirectional image data 302 First marker detection processing unit 303 Marker information 304 Omnidirectional camera marker posture coordinate information 305 Omnidirectional camera position estimation processing unit 306 Omnidirectional camera coordinate information 307 Goggle area extraction processing unit 308a 1st goggle area image data 308b 2nd goggle area image data 308n n-th goggle area image data 309 2nd marker detection processing unit 310a 1st goggle marker attitude coordinate information 310b 2nd goggles Marker attitude coordinate information 310n... n-th goggle marker attitude coordinate information 311... goggle coordinate estimation processing unit 312a... first goggle attitude coordinate information 312b... second goggle attitude coordinate information 312n... n-th goggle attitude coordinate information 313 Goggles image generation processing unit 314a First goggles viewpoint image data 314b Second goggles viewpoint image data 314n n-th goggles viewpoint image data 315 Three-dimensional shape estimation processing unit 316 Virtual three-dimensional image Data 401 Subject 501 Passenger car 502 Wide-angle camera 503 Wall 504 Child 505 Road reflector 506 Road marking 601 Video processing system 602 First marker 603 Second Two markers 604 Video processing device 701 First marker information 702 Second marker information

Claims (9)

a reflector that reflects any target;
a camera capable of photographing any object reflected by the reflector;
a first marker present at a position directly photographable from the camera;
a second marker whose mirror image reflected by the reflector can be captured by the camera;
The space is grasped from the position of the first marker present in the image data output by the camera, the image portion of the reflector is cut out from the image data, and the second marker included in the image portion of the reflector is extracted. and an image processing device that recognizes distortion caused by the reflector from its shape and corrects the image portion of the reflector to a flat state. the reflector is a convex reflector,
In the mirror image obtained from the convex reflector, the image processing device corrects the geometric distortion of the mirror image caused by the shape of the convex reflector, thereby flattening the image portion of the convex reflector. 2. The image processing system of claim 1, wherein the image processing system corrects. The video processing device is
a first marker detection processing unit that finds the first marker appearing in the video data and outputs camera marker posture coordinate information;
a camera position estimation processing unit that performs estimation calculation processing based on the camera marker posture coordinate information and outputs camera coordinate information that is coordinate information in the virtual three-dimensional space of the camera;
a reflector region extraction processing unit that outputs reflector region video data obtained by extracting one or more of the reflector portions from the video data;
A second step of finding a mirror image of the second marker appearing in one or more of the reflector area image data and outputting reflector marker attitude coordinate information, which is position information and attitude information in the one or more of the reflector area image data. a marker detection processing unit;
Based on the one or more reflector marker orientation coordinate information, the camera marker orientation coordinate information, and the camera coordinate information, the second image captured by the one or more reflectors in the virtual three-dimensional space of the camera is displayed. a reflector coordinate estimation processing unit that outputs reflector orientation coordinate information that is information including the position, orientation, and distortion of the marker;
Reflection estimated as the direction in which the reflector faces, based on one or more of the reflector region image data, one or more of the reflector orientation coordinate information, the camera marker orientation coordinate information, and the camera coordinate information. Body viewpoint image data is generated, and geometric distortion of the reflector area image data caused by the shape of the convex reflector is corrected when the reflector area image data is image data by the convex reflector. 3. The image processing system according to claim 2, further comprising a reflector image generation processing unit that corrects the image portion of the convex reflector to a planar state.
The video processing device further includes:
4. The image processing system according to claim 3, further comprising a three-dimensional shape estimation processing unit that generates virtual three-dimensional image data of a subject based on a plurality of said reflector viewpoint image data. 5. The image processing system of claim 2, 3 or 4, wherein the reflector is goggles. 5. An image processing system according to claim 2, 3 or 4, wherein said reflector is a reflector. 7. A video processing system according to claim 5 or 6, wherein said first marker and said second marker are identical. a reflector that reflects an arbitrary object; a camera capable of photographing the arbitrary object reflected by the reflector; a first marker that can be photographed from the reflector, a second marker that can be photographed by the camera as a mirror image reflected by the reflector, and the position of the first marker that exists in the image data output by the camera. Then, an image portion of the reflector is cut out from the image data, distortion caused by the reflector is grasped from the shape of the second marker included in the image portion of the reflector, and the image portion of the reflector is obtained. A video processing device in a video processing system comprising a video processing device that corrects to a planar state,
The video processing device is
a first marker detection processing unit that finds the first marker appearing in the video data and outputs camera marker posture coordinate information;
a camera position estimation processing unit that performs estimation calculation processing based on the camera marker posture coordinate information and outputs camera coordinate information that is coordinate information in the virtual three-dimensional space of the camera;
a reflector region extraction processing unit that outputs reflector region video data obtained by extracting one or more of the reflector portions from the video data;
A second step of finding a mirror image of the second marker appearing in one or more of the reflector area image data and outputting reflector marker attitude coordinate information, which is position information and attitude information in the one or more of the reflector area image data. a marker detection processing unit;
Based on the one or more reflector marker orientation coordinate information, the camera marker orientation coordinate information, and the camera coordinate information, the second image captured by the one or more reflectors in the virtual three-dimensional space of the camera is displayed. a reflector coordinate estimation processing unit that outputs reflector orientation coordinate information that is information including the position, orientation, and distortion of the marker;
Reflection estimated as the direction in which the reflector faces, based on one or more of the reflector region image data, one or more of the reflector orientation coordinate information, the camera marker orientation coordinate information, and the camera coordinate information. Generating body viewpoint image data, and correcting geometric distortion of the reflector area image data caused by the shape of the convex reflector when the reflector area image data is image data of a convex reflector. and a reflector image generation processing unit that corrects the image portion of the convex reflector to a flat state. Furthermore,
9. The image processing apparatus according to claim 8, further comprising a three-dimensional shape estimation processing unit that generates virtual three-dimensional image data of a subject based on a plurality of said reflector viewpoint image data.

Images