JP4283794B2 - Monitoring device - Google Patents

Description

  The present invention relates to a monitoring device, and more particularly to a monitoring device that displays a video of a location desired by a user from a wide range of surrounding video.

  Conventionally, a monitoring apparatus that switches and displays a desired video from two or more video has been proposed.

  For example, in the device described in Patent Document 1, when the lift position of the fork is equal to or lower than a predetermined first lift position, a front image of the vehicle captured by the traveling camera is displayed on the display device, and the lift height of the fork is displayed. When the position exceeds the predetermined first lift position, a front image of the fork captured by the cargo handling camera is displayed on the display device.

Moreover, in the apparatus described in Patent Document 2, the operator's head position detected by the head position detection sensor is supplied to the 3D image input apparatus as head position data. The actuator in the three-dimensional image input device rotates the axes x, y, and z according to the head position data. As each axis of the actuator rotates, the imaging direction of the TV camera of the three-dimensional image input device changes. New image data captured by the TV camera is wirelessly output to the dome type display device.
JP 2003-212494 A JP-A-6-339153

  However, in the apparatus described in Patent Document 1, only the front image of the vehicle or the front image of the fork is displayed. When the vehicle turns right, an image in the right direction is required, and when turning left, an image in the left direction is required. However, the apparatus described in Patent Document 1 cannot display these images.

  In Patent Document 2, since it is necessary to move the camera in order to switch the video, a controller for driving the camera is required.

  Therefore, an object of the present invention is to provide a monitoring device that can switch and display a desired image from a wide range of surrounding images without requiring camera control.

  In order to solve the above problems, a monitoring apparatus according to the present invention includes one or a plurality of cameras for photographing the surroundings, a surrounding image generating unit that generates a wide range of surrounding images by photographing with the cameras, and a user A detection unit that detects the position, state, or operation of the display, a display video generation unit that generates a display video according to the detection result from the generated surrounding video, and a display unit that displays the generated display video Is provided.

  Preferably, the user's field of view is obstructed by an object, the camera is disposed at a position for photographing outside the user's field of view, and the display unit or a screen for projection by the display unit is disposed within the user's field of view. .

  Preferably, the display unit or the screen for projection by the display unit is disposed on a surface of the object that blocks the user's field of view within the user's field of view, or at a location other than the surface of the object within the user's field of view.

  Preferably, the detection unit detects the position of the user's face, and the display video generation unit generates a display video according to the detected position of the face.

  Preferably, the detection unit further detects the distance between the display unit or the screen for projection by the display unit and the face, and the display image generation unit generates the generated surrounding image as the detected distance is shorter A display image that includes a wider range of images is generated.

  Preferably, the face part detected by the detection part is at least one of the user's face, eyes, nose, eyebrows, and mouth.

  Preferably, the detection unit detects the direction of the user's line of sight, and the display video generation unit generates a display video according to the detected line of sight.

  Preferably, the monitoring device is a device that monitors the surroundings of the vehicle, the camera is disposed outside the vehicle, and the display unit or the screen for projection by the display unit is disposed around the driver's seat of the vehicle. The

  Preferably, the monitoring device is a device that monitors one of the two regions partitioned by the wall, and the camera is disposed in one region of the two regions partitioned by the wall. The display or the screen for projection by the display is attached to the wall in the other of the two regions partitioned by the wall.

  According to the monitoring apparatus according to an aspect of the present invention, it is possible to switch and display a desired image from a wide range of surrounding images without requiring complicated control. And even if there is an object that obstructs the field of view, it is possible to see the other side of the object as if there is no object.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
1st Embodiment is related with the monitoring apparatus which produces | generates the image | video for a display according to the position between eyebrows from the surrounding image | video.

(Configuration of monitoring device)
FIG. 1 is a functional block diagram of the monitoring apparatus according to the first embodiment.

  With reference to FIG. 1, the monitoring device 1 includes an omnidirectional camera 11, an ambient video generation unit 12, a detection unit 18, a display video generation unit 81, and a display unit 16. The detection unit 18 includes a face photographing camera 13 and an eyebrow position detection unit 14.

  The display unit 16 is a liquid crystal display, for example, and displays the generated display video. Hereinafter, the display unit 16 may be referred to as a display.

  The omnidirectional camera 11 includes a hyperboloidal mirror and a camera, reflects light reaching from 360 degrees around the curved mirror, and shoots the reflected light with the camera.

  The surrounding video generation unit 12 generates a circular video (round video) from the data captured by the omnidirectional camera 11 and outputs the circular video to the display video generation unit 81.

The face photographing camera 13 photographs a user's face and generates a face image.
FIG. 2 is a diagram illustrating an example in which the monitoring device 1 according to the first embodiment is attached to a tank which is an example of a vehicle.

  Referring to FIG. 2, the omnidirectional camera 11 is arranged at a position where the user (driver) takes a picture outside the field of view, that is, outside the tank.

  The display part 16 is arrange | positioned around the driver's seat which is in a user's (driver | operator's) visual field. That is, the display unit 16 includes a front display 51 disposed in front of the driver seat, a rear display 52 disposed behind the driver seat, a right side display 53 disposed on the right side of the driver seat, and a left side of the driver seat. It consists of a left side display (not shown) arranged.

  The face photographing camera 13 is attached to the upper part of each display. The horizontal direction of the mounting position of the face photographing camera 13 is preferably the center in the horizontal direction of the display.

  Other components of the monitoring device 1 are arranged inside the tank or at another place. These components are connected to the face photographing camera 13, the omnidirectional camera 11, and the display unit 16 by wireless or wired connection.

  The eyebrow position detecting unit 14 detects the position between the eyebrows from the face image. As a method for detecting the position between the eyebrows, for example, the method of Japanese Patent No. 3405511 is used. According to this method, digital data of the value of each pixel in a target image area that is a human face area is prepared, and each pixel in the target pixel area is preliminarily surrounded by the pixel. The relative amount of the specific wave number component indicated by the distribution curve of the pixel values on the defined closed curve is obtained, and the face where the relative amount of the specific wave number component is locally maximum in the face area is determined. It is extracted as a point representing the space between eyebrows as a feature point in the region.

  3A to 3C are diagrams illustrating an example of detecting the position between the eyebrows. These figures are face images photographed by the face photographing camera 13, and the inter-eyebrow position detection unit 14 detects the inter-brow position based on these face images.

  FIG. 3A shows a case where the position between the eyebrows is at the center in the horizontal direction of the face image. In this case, the horizontal relative positional relationship between the display and the eyebrows is detected when the eyebrows are at the center of the display toward the display.

  FIG. 3B shows a case where the position between the eyebrows is on the right side of the face image in the horizontal direction. In this case, the horizontal relative positional relationship between the display and the eyebrows is detected when the eyebrows are on the left side of the center of the display toward the display.

  FIG. 3C shows the case where the position between the eyebrows is on the left side of the face image in the horizontal direction. In this case, the horizontal relative positional relationship between the display and the eyebrows is detected when the eyebrows are on the right side of the center of the display toward the display.

  The display video generation unit 81 generates a display video corresponding to the position of the user's eyebrows from the surrounding video. The display video generation unit 81, the conversion map generation unit 17, the conversion map memory 19, Conversion unit 15.

The ambient video memory 10 stores the ambient video output from the ambient video generation unit 12.
The conversion map generation unit 17 creates a conversion map for converting the surrounding video into a display video according to the following procedure, and stores it in the conversion map memory 19.

  First, the conversion map generation unit 17 specifies a portion of the surrounding video that should be included in the display video according to the detected position between the eyebrows.

  That is, the conversion map generation unit 17 causes the display video to include the surrounding video of the portion on the right side of the display as the detected position between the eyebrows is on the left side of the display center. Identify the surrounding video included in the video for display. By doing so, it is possible to view a surrounding image corresponding to the peeping operation.

  In addition, the conversion map creating unit 17 includes the display image including the surrounding image of the portion on the left side of the display as the position between the detected eyebrows is on the right side of the display center. Identify the surrounding video included in the video for display. By doing so, it is possible to view a surrounding image corresponding to the peeping operation.

  4A to 4C are diagrams showing specific examples of the relationship between the position between the eyebrows and the portion of the surrounding video specified to be included in the display video.

  4A to 4C, the distance between the eyebrows and the display is a. Here, a is assumed to be a predetermined value. Let b0 and b1 be the distance from the point where the perpendicular line dropped from the position between the eyebrows to the display to the left end of the display and the right end of the display, respectively. When the horizontal width of the display is b, b0 + b1 = b.

An image for display is a peripheral image in the left direction between a vertical line and a boundary shifted counterclockwise by an angle α0 from the normal between the eyebrows, an angle α1 from the vertical and the vertical lines centered on the eyebrows It is assumed that the image includes a surrounding image in the right direction within a range sandwiched by a boundary shifted clockwise only. Then, α0 = tan ⁻¹ (b0 / a) and α1 = tan ⁻¹ (b1 / a) are established. α = α0 + α1. As b0 is larger, α0 is larger, and as b1 is larger, α1 is larger.

  In FIG. 4A, b0 = b1 since the position between the eyebrows is at the center of the display toward the display. In this case, α0 = α1.

  In FIG. 4B, the position between the eyebrows is on the left side of the center of the display toward the display, so b0 <b1. In this case, since α0 <α1, the surrounding image of the portion on the right side toward the display is included rather than when the position between the eyebrows is at the center of the display.

  In FIG. 4C, b0> b1 because the position between the eyebrows is on the right side of the display center toward the display. In this case, since α0> α1, the surrounding image of the portion on the left side toward the display is included rather than when the position between the eyebrows is in the center of the display.

  The conversion map generation unit 17 calculates a primary conversion formula for enlarging and / or reducing the surrounding video portion specified to be included in the display video so as to fit the size of the display area of the display unit 16 (display). calculate. Further, the conversion map generation unit 17 calculates a primary conversion formula for panoramic development of a portion of the surrounding video specified to be included in the display video.

  The conversion map generation unit 17 calculates a primary conversion expression obtained by combining the two primary conversion expressions, and creates a conversion map according to the combined primary conversion expression. This conversion map defines each pixel of the surrounding image (circular image) corresponding to each pixel of the display image.

  The conversion unit 15 generates a display video by rearranging the pixels of the surrounding video in the surrounding video memory 10 according to the conversion map in the conversion map memory 19.

(Operation of the monitoring device 1)
FIG. 5 is a flowchart showing an operation procedure of the monitoring apparatus 1 according to the first embodiment.

  First, the omnidirectional camera 11 captures light reaching from 360 degrees around (step S101).

  Next, the surrounding image generation unit 12 generates a circular image as the surrounding image from the data obtained by the omnidirectional camera 11, and stores it in the surrounding image memory 10 (step S102).

  The following steps S103 to S105 are performed in parallel with the processing of the above steps S101 to S102.

The face photographing camera 13 shoots a face and generates a face image (step S103).
Next, the position between the eyebrows is detected from the face image (step S104).

  Next, the conversion map generation unit 17 creates a conversion map for converting the surrounding video into the display video according to the position between the user's eyebrows, and stores it in the conversion map memory 19 (step S105).

  After both steps S102 and S105 are completed, the conversion unit 15 converts the surrounding image in the surrounding image memory 10 into a display image according to the conversion map in the conversion map memory 19 (step S106).

Next, the display unit 16 displays the generated display video (step S107).
As described above, according to the monitoring device 1 of the first embodiment, without changing the camera control, the user changes the display image so as to include a desired portion around by changing the position between the eyebrows. Can be made. By using this monitoring device 1 for a tank, the driver of the tank moves the face position (the position between the eyebrows), that is, by looking into the outside through the window, so that the surrounding image of the outside of the tank can be displayed. Can see.

[Second Embodiment]
The second embodiment relates to a monitoring device that generates a display image corresponding to a line-of-sight direction from surrounding images.

(Configuration of monitoring device)
FIG. 6 is a functional block diagram of the monitoring apparatus according to the second embodiment.

  The difference between the monitoring device 2 of FIG. 6 and the monitoring device 1 of the first embodiment of FIG. 1 will be described.

  The detection unit 28 of the monitoring device 2 in FIG. 6 includes a line-of-sight direction detection unit 24 instead of the inter-brow position detection unit 14.

  The gaze direction detection unit 24 detects the gaze direction of the user. For the method of detecting the gaze direction, refer to the document “Estimation of gaze from a monocular camera based on four reference points and three calibration images, Shinjiro Kawado, Akira Utsumi, Shinji Abe, Image Recognition and Understanding Symposium (MIRU2005), 1337 to 1341, July 2005 ”is used. This method is a method for estimating the line-of-sight direction with a monocular camera using four reference points fixed on the face and three calibration images. In this method, first, a parameter for estimating the projection position of the center of the eyeball is extracted from the calibration image by linear combination of the coordinates of the four reference points. A vector connecting the projection center of the eyeball center and the iris center is estimated as the line-of-sight direction. For more details of this method, please refer to the above-mentioned document.

  The display video generation unit 82 generates a display video corresponding to the user's line-of-sight direction from the surrounding video. The display video generation unit 82, the surrounding video memory 10, the conversion map generation unit 27, the conversion map memory 19, and the conversion Part 15. The surrounding video memory 10, the conversion map memory 19, and the conversion unit 15 are the same as those in the first embodiment.

  The conversion map generator 27 creates a conversion map for converting the surrounding video into the display video according to the user's line-of-sight direction according to the following procedure, and stores it in the conversion map memory 19.

  First, the conversion map generation unit 2 specifies a portion of the surrounding video that should be included in the display video according to the detected line-of-sight direction. In other words, the conversion map generation unit 17 specifies that the surrounding video within the range of the angle α centered on the center line of the user's face (the line passing through the eyebrows and perpendicular to the face) is included in the display video. To do. The direction of the center of this range (the direction of the line passing through the center of this range) is the direction of the detected line of sight. Here, α is a predetermined value. The center of the user's face is assumed not to move.

  FIGS. 7A to 7C are diagrams showing specific examples of the relationship between the line-of-sight direction and the portion of the surrounding video specified as being included in the display video.

  In FIG. 7A, since the user's line-of-sight direction is in the center of the display, the display video includes a surrounding video in front of the display.

  In FIG. 7B, since the user's line-of-sight direction is on the right side of the center of the display, the display image includes a surrounding image on the right front side of the display.

  In FIG. 7C, since the user's line-of-sight direction is on the left side of the center of the display, the display image includes a surrounding image on the left front of the display.

  The conversion map generation unit 27 calculates a primary conversion formula for enlarging and / or reducing the surrounding video portion specified to be included in the display video so as to fit the size of the display area of the display unit 16 (display). To do. Furthermore, the conversion map generation unit 27 calculates a primary conversion formula for panoramic development of a portion of the surrounding video specified to be included in the display video.

  The conversion map generation unit 27 calculates a primary conversion formula obtained by combining the two primary conversion formulas, and creates a conversion map according to the combined primary conversion formula. This conversion map defines each pixel of the surrounding image (circular image) corresponding to each pixel of the display image.

(Operation of monitoring device 2)
FIG. 8 is a flowchart showing an operation procedure of the monitoring device 2 according to the second embodiment.

  Referring to FIG. 8, steps S101 to S103 are the same as the operation procedure of the first embodiment of FIG.

  In step S204, the gaze direction detection unit 24 detects the gaze direction of the user from the face image.

  Next, in step S <b> 205, the conversion map generation unit 27 creates a conversion map for converting the surrounding video into the display video according to the user's line-of-sight direction, and stores it in the conversion map memory 19.

  Subsequent steps S106 and 107 are the same as the operation procedure of the first embodiment in FIG.

  As described above, according to the monitoring device 2 of the second embodiment, it is possible to change the display image so as to include a desired peripheral portion by changing the line-of-sight direction without requiring camera control. it can.

[Third Embodiment]
The third embodiment relates to a monitoring device that generates a display video according to the distance from the surrounding video to the eyes and the display.

(Configuration of monitoring device)
FIG. 9 is a functional block diagram of the monitoring apparatus according to the third embodiment.

  The difference between the monitoring device 3 of FIG. 9 and the monitoring device 1 of the first embodiment of FIG. 1 will be described.

  The detection unit 38 of the monitoring device 3 in FIG. 9 includes a distance detection unit 34 instead of the inter-brow position detection unit 14.

  The distance detection unit 34 detects the distance between the user's eyes (left eye or right eye) and the face photographing camera 13 using the face image. The distance detection unit 34 detects a dark region at a target position on both sides of the eyebrows detected by the method described in the first embodiment as an eye region, and measures the size (for example, the number of pixels) of the eye region. Then, the relative distance between the eyes and the face photographing camera 13 is detected by comparing the measured size of the eye area with the reference value. For example, when the reference value of the size of the eye region is S1, the distance at that time is d1, and the size of the measured eye region is St, the distance is detected as dt = d1 × St / S1. The distance dt is further converted into the distance between the eyes and the display on a constant basis.

  The display video generation unit 83 generates a display video corresponding to the distance between the user's eyes and the display from the surrounding video. The display video generation unit 83, the conversion map generation unit 37, and the conversion map memory 19 and a conversion unit 15. The surrounding video memory 10, the conversion map memory 19, and the conversion unit 15 are the same as those in the first embodiment.

  The conversion map generation unit 37 creates a conversion map for converting the surrounding video into a display video according to the following procedure, according to the distance between the user's eyes and the display, and stores the conversion map in the conversion map memory 19.

  First, the conversion map generation unit 37 specifies a portion of the surrounding video that should be included in the display video according to the distance between the eyes and the display.

That is, the conversion map generation unit 37 displays the surrounding video within the range of the angle θ centered on the center line of the user's face (line passing between the eyebrows) with the front direction of the display (direction toward the display) as the center. Specify to be included in the video. Here, the angle θ is θ = 2 × tan ⁻¹ (b / (2 × a)), where a is the viewing distance and b is the horizontal length of the display. Here, it is assumed that the center of the user's face is fixed at the center of the display toward the display. The shorter the distance a, the larger the angle θ, and the wider the range of surrounding video included in the display video. In this case, the range of the surrounding video included in the display video may be narrowed as the distance a is short, and the range of the surrounding video included in the display video may be widened as the distance a is long.

  FIGS. 10A to 10C are diagrams showing specific examples of the relationship between the distance between the eyes and the display and the portion of the surrounding video specified as being included in the display video.

  In FIG. 10B, since the distance a between the eyes and the display is shorter than in FIG. 10A, the angle θ is larger than in FIG. 10A, and the display image has a wider range of surroundings. Includes video.

  In FIG. 10C, since the distance a between the eyes and the display is longer than in FIG. 10A, the angle θ is smaller than in FIG. 10A, and the display image is in a narrower range. Includes ambient video.

  The conversion map generation unit 37 calculates a primary conversion formula for enlarging and / or reducing a portion of the surrounding video specified to be included in the display video so as to fit the size of the display area of the display unit 16 (display). To do. Furthermore, the conversion map generation unit 37 calculates a primary conversion formula for panoramic development of a portion of the surrounding video specified to be included in the display video.

  The conversion map generation unit 37 calculates a primary conversion formula obtained by combining the two primary conversion formulas, and creates a conversion map according to the combined primary conversion formula. This conversion map defines each pixel of the surrounding image (circular image) corresponding to each pixel of the display image.

(Operation of the monitoring device 3)
FIG. 11 is a flowchart showing an operation procedure of the monitoring device 3 according to the third embodiment.

  Referring to FIG. 11, steps S101 to S103 are the same as the operation procedure of the first embodiment of FIG.

  In step S304, the distance detection unit 34 detects the distance between the user's eyes and the display unit 16 (display) using the face image.

  Next, in step S305, the conversion map generation unit 37 creates a conversion map for converting from surrounding video to display video according to the distance between the user's eyes and the display unit 16 (display), and the conversion map. The data is stored in the memory 19.

  Subsequent steps S106 and 107 are the same as the operation procedure of the first embodiment in FIG.

  As described above, according to the monitoring device 3 of the third embodiment, without changing the camera control, the distance between the user's eyes and the display is changed so that a desired surrounding portion is included. The video can be changed.

(Modification)
The present invention is not limited to the above embodiment, and includes, for example, the following modifications.

(1) See-through display A so-called see-through display (pseudo window) that can be seen on the surface of an uneven surface as if there is no other wall on the uneven wall surface using the monitoring device of the embodiment of the present invention. Can be realized.

(Monitoring outside the room)
FIG. 12 is a diagram illustrating an example of attachment of each component when a see-through display is realized using the monitoring device according to the embodiment of the present invention.

  FIG. 12 shows an example of attachment in the case of monitoring the outside of a room which is one of two areas partitioned by a wall. It is assumed that the user exists in the room which is the other of the two areas partitioned by the wall.

  Referring to FIG. 12, the omnidirectional camera 11 is attached to a position where an image is taken outside the user's field of view, that is, outside the room being monitored. The projector 54, which is an example of the display unit 16, is attached to a position in the user's visual field, that is, the ceiling inside the room. A part of the inner wall of the room is formed as a screen 55 for projection by the projector 54.

  The face photographing camera 13 is attached to the inner wall of the room. The vertical direction of the attachment position of the face photographing camera 13 is above or below the screen 55, and the horizontal direction of the attachment position of the face photographing camera 13 is the center of the horizontal direction of the screen 55.

  The other components of the monitoring device are placed inside the room or elsewhere. These components are connected to the face photographing camera 13, the omnidirectional camera 11, and the projector 54 by wireless or wired connection.

(Monitoring inside the reactor chamber)
FIG. 13 is a diagram illustrating another example of attachment of each component when a see-through display is realized using the monitoring device according to the embodiment of the present invention.

  FIG. 13 shows an example of attachment in the case of monitoring the inside of a room (reactor room) in which a nuclear reactor, which is one of two areas partitioned by walls, is placed. The user is assumed to exist outside the reactor chamber, which is the other of the two regions partitioned by the wall.

  Referring to FIG. 13, the omnidirectional camera 11 is attached to a position outside the user's field of view, that is, inside the reactor chamber to be monitored. The projector 54, which is an example of the display unit 16, is attached to a position in the user's field of view, that is, a ceiling outside the reactor room. A part of the outer wall of the reactor chamber is formed as a screen 55 for projection by the projector 54.

  The face photographing camera 13 is attached to the outer wall of the reactor chamber. The vertical direction of the attachment position of the face photographing camera 13 is above or below the screen 55, and the horizontal direction of the attachment position of the face photographing camera 13 is the center of the horizontal direction of the screen 55.

  Other components of the monitoring device are located outside the reactor room or elsewhere. These components are connected to the face photographing camera 13, the omnidirectional camera 11, and the projector 54 by wireless or wired connection.

  In the examples of FIGS. 12 and 13, the size of the image displayed on the screen may be the same as the size of the actual object. Thereby, the screen can be made closer to the window.

(2) About eyebrow detection, eye gaze direction detection and distance detection between eyes and display The methods of eyebrow detection, eye gaze direction detection and distance detection between eyes and display described in the embodiment of the present invention are examples. However, the present invention is not limited to this, and other methods may be used.

  For example, the method described in Japanese Patent Application Laid-Open No. 2004-185611 may be used for detecting the position between the eyebrows.

  For the detection of the gaze direction, the method described in the document “Interface Using Gaze, Takehiko Ohno, Information Processing (Vol.44, No.7), pp.726-732, July 2003” may be used.

  For the detection of the distance between the eyes and the display, a method of measuring the distance between human eyes or a method using a two-lens stereo camera, or a method using a distance sensor using ultrasonic waves or the like may be used.

(3) Regarding the feature points of the face other than the eyebrows and eyes In the first embodiment of the present invention, the eyebrows are used as the feature points of the face part (all or part of the face), but the present invention is not limited to this. In the same way as detecting other feature points of the face and generating a display image according to the detected position between the eyebrows, a display image is generated according to the center point of the detected feature point. Also good.

  In the third embodiment of the present invention, the eyes are used as the facial feature points when detecting the distance to the display. However, the present invention is not limited to this, and other facial feature points are detected. Then, it is also possible to generate a display image corresponding to the detected feature point and the distance to the display in the same manner as generating the display image corresponding to the distance between the eyes and the display.

  As the facial feature point, at least one of the entire face, left eye, right eye, nose, eyebrows, and mouth can be used.

  For example, the method described in Japanese Patent Application Laid-Open No. 2004-185611 can be used for position detection of the entire face, and the method described in Japanese Patent Application Laid-Open No. 2003-168121 can be used for position detection of the eyes. For detecting the position of the nose, for example, the method described in Japanese Patent Application Laid-Open No. 2004-157778 can be used, and for detecting the position of the mouth, for example, the documents “Shinjiro Kawato, Akira Utsumi, Kazuhiro Kuwahara, Use the method described in the Symposium on Image Recognition and Understanding (MIRU2004), I-107-108, July 2004, which uses real-time image processing as the interface for the eyes, nose, and mouth as input devices. Can do.

(4) Regarding generation of display video from surrounding video In the embodiment of the present invention, a panoramic display video is generated by first converting a circular video surrounding video. However, the present invention is not limited to this. Alternatively, a circular video may be converted into a primary image to generate a panoramic video, a partial video may be cut out from the panoramic video, and the partial video may be enlarged and / or reduced to generate a display video.

(5) Enlarging and / or reducing In the embodiment of the present invention, a part of the surrounding video included in the display video is enlarged and / or reduced to fit the size of the display. However, the present invention is not limited to this. For example, a part of the surrounding video may be always enlarged and / or reduced at a certain ratio, and may be displayed on the display without being enlarged or reduced.

  In the third embodiment, from the surrounding video, a part of the surrounding video centered on the direction of the line of sight is enlarged and / or reduced to the size of the display and displayed in the entire display area of the display. As shown in FIG. 14, the display image may be displayed only on a specific portion of the display area of the display or screen (in the case of a projector), for example, only in the vicinity of the line-of-sight direction.

(6) Regarding Input from User In the embodiment and the modification of the present invention, from the surrounding video according to the position of the user's face, the distance between the face and the display, the line of sight, and the operation of the user's hand or foot. Although the display video is generated, the present invention is not limited to this. The display video can be used from the surrounding video according to at least one of the user's position, state, and action. For example, the position of the entire body of the user may be used as the position of the user. Further, the angle between the normal of the surface approximating the entire face of the user and the display may be used as the user state. Further, as the user's operation, a user's voice or gesture may be used.

(7) Camera for Capturing Ambient Image In the embodiment of the present invention, the omnidirectional camera 11 is used to generate a wide range of ambient images. However, the present invention is not limited to this. A wide range of surrounding images may be generated from images captured by a plurality of cameras. Further, in the embodiment of the present invention, the surrounding video of 360 degrees around is generated, but any wide angle video as long as the user needs may be generated. An infrared camera may be used.

(8) About arrangement of omnidirectional camera In the embodiment and the modification of the present invention, the omnidirectional camera is arranged at a position for photographing outside the user's field of view. This is because it is considered that there is no meaning to display the surrounding image on the display within the user's field of view. If the condition that the omnidirectional camera itself is arranged outside the user's field of view is satisfied as in the embodiment and the modification example (1) of the present invention, the surrounding image captured by the omnidirectional camera is inevitably also captured by the user. If it is outside the field of view, it is only necessary to consider satisfying the condition that the omnidirectional camera is placed outside the user's field of view.

(9) Specific Example of Display Unit In the embodiment of the present invention, a liquid crystal display is used as the display unit 16 and a projector is used in the modification (1). However, the present invention is not limited to this. As the display unit 16, for example, a CRT (Cathode Ray Tube) or a plasma display may be used. In the embodiment of the present invention, when a projector is used as the display unit, a screen for projection by the projector is formed at the position of the display. In the third embodiment, the distance detection unit detects the distance between the screen and the user's eyes. Further, in the case of a vehicle, a screen may be provided on a part of the windshield.

(10) Correction Processing In the embodiment of the present invention, when displaying an image around the center line of the face, for example, in the range of θ0 <θ <θ1, the range of θ0 <θ <θ1 of the surrounding image is displayed. However, it is assumed that the space between the user's eyebrows is at the position of the omnidirectional camera 11. Actually, since the position between the user's eyebrows and the position of the omnidirectional camera 11 are different, the display range may be corrected based on the difference between these positions.

(11) Other Application Examples The monitoring device according to the embodiment of the present invention can be used for uses other than those described above. For example, monitoring the front and rear of special vehicles other than tanks, monitoring from inside the refrigerator, freezer, dangerous goods garage, etc., monitoring the cabin from the pilot room of the airplane, monitoring the pilot room from the airplane cabin, manhole, It can be used for external monitoring such as in a basement, monitoring disabled persons, and blind spots in offices and factories.

(12) About creation of conversion map About the creation of a conversion map, it is good also as what combines parse conversion and rotation conversion further.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a functional block diagram of the monitoring apparatus of 1st Embodiment. It is a figure showing the example which attached the monitoring apparatus 1 of 1st Embodiment to the tank which is an example of a vehicle. (A)-(c) is a figure showing the example of a detection of the position between eyebrows. (A)-(c) is a figure showing the specific example of the relationship between the position between eyebrows, and the part of the surrounding image specified so that it may be included in the image | video for a display. It is a flowchart showing the operation | movement procedure of the monitoring apparatus of 1st Embodiment. It is a functional block diagram of the monitoring apparatus of 2nd Embodiment. (A)-(c) is a figure showing the specific example of the relationship between a gaze direction and the part of the surrounding image identified as being contained in the image | video for a display. It is a flowchart showing the operation | movement procedure of the monitoring apparatus of 2nd Embodiment. It is a functional block diagram of the monitoring apparatus of 3rd Embodiment. (A)-(c) is a figure showing the specific example of the relationship between the distance of eyes and a display, and the part of the surrounding image specified as being included in the image | video for a display. It is a flowchart showing the operation | movement procedure of the monitoring apparatus of 3rd Embodiment. It is a figure showing the example of attachment of each component in the case of implement | achieving a see-through display using the monitoring apparatus of embodiment of this invention. It is a figure showing another example of attachment of each component in the case of implement | achieving a see-through display using the monitoring apparatus of embodiment of this invention. It is a figure showing the example of a display of the image | video for a display.

Explanation of symbols

  1, 2, 3 Monitoring device, 10 Ambient image memory, 11 Omnidirectional camera, 12 Ambient image generation unit, 13 Face shooting camera, 14 Eyebrow position detection unit, 15, 25, 35 Conversion unit, 16 Display unit, 17 , 27, 37 Conversion map generation unit, 18, 28, 38 detection unit, 19 conversion map memory, 24 gaze direction detection unit, 34 distance detection unit, 51 front display, 52 rear display, 53 right side display, 54 projector, 55 Screen, 81, 82, 83 Display image generation unit.

Claims (8)

One or more cameras for photographing the surroundings;
A surrounding image generation unit that generates a wide range of surrounding images by shooting with the camera;
A peripheral video memory for storing the peripheral video generated by the peripheral video generator;
A detection unit for detecting the position of the user's face;
In accordance with the position of the face detected by the detection unit, a part of the surrounding video stored in the surrounding video memory is specified, and a display video that generates the display video including the specified part is generated. A video generator;
A display unit for displaying the generated display video,
The display video generation unit is configured so that the position of the face detected by the detection unit is located on the left side of the center of the screen for projection by the display unit or the display unit. Generating a display image including a surrounding image on the right side of the display unit or a screen for projection by the display unit, and the position of the face detected by the detection unit is The closer to the right side of the center of the screen for the projection by the display unit or the display unit, the more the left side of the generated surrounding image toward the screen for the projection by the display unit or the display unit. Generate a display image that includes a certain surrounding image ,
The display video generation unit
A conversion map creating unit that creates a conversion map that defines pixels of the surrounding video corresponding to each pixel of the display video;
A conversion map memory for storing the created conversion map;
A conversion unit that generates the display video by rearranging the pixels in the surrounding video memory according to the conversion map stored in the conversion map memory;
The conversion map creating unit specifies a part of the surrounding video, calculates a primary conversion formula for correcting the part of the specified surrounding video to fit the size of the display area of the display unit, A monitoring device that creates the conversion map based on the primary conversion formula . One or more cameras for photographing the surroundings;
A surrounding image generation unit that generates a wide range of surrounding images by shooting with the camera;
A peripheral video memory for storing the peripheral video generated by the peripheral video generator;
A detection unit for detecting the position of the user's face;
In accordance with the position of the face detected by the detection unit, a part of the surrounding video stored in the surrounding video memory is specified, and a display video that generates the display video including the specified part is generated. A video generator;
A display unit for displaying the generated display video,
The detection unit detects a distance between the display unit or the screen for projection by the display unit and the face unit,
The display image generation unit generates a display image that includes a wider range of images in the generated surrounding image as the distance detected by the detection unit is shorter ,
The display video generation unit
A conversion map creating unit that creates a conversion map that defines pixels of the surrounding video corresponding to each pixel of the display video;
A conversion map memory for storing the created conversion map;
A conversion unit that generates the display video by rearranging the pixels in the surrounding video memory according to the conversion map stored in the conversion map memory;
The conversion map creating unit specifies a part of the surrounding video, calculates a primary conversion formula for correcting the part of the specified surrounding video to fit the size of the display area of the display unit, A monitoring device that creates the conversion map based on the primary conversion formula .   The monitoring device according to claim 1 or 2, wherein the detection unit detects at least one of the user's face, eyes, nose, eyebrows, and mouth. The monitoring device is a device for monitoring the surroundings of a vehicle,
The camera is arranged outside the vehicle;
The monitoring device according to claim 1, wherein the display unit or a screen for projection by the display unit is arranged around a driver's seat of the vehicle. A monitoring device that monitors one of two regions partitioned by a wall,
One or more cameras for photographing the surroundings;
A surrounding image generation unit that generates a wide range of surrounding images by shooting with the camera;
A peripheral video memory for storing the peripheral video generated by the peripheral video generator;
A detection unit for detecting the position of the user's face;
In accordance with the position of the face detected by the detection unit, a part of the surrounding video stored in the surrounding video memory is specified, and a display video that generates the display video including the specified part is generated. A video generator;
A display unit for displaying the generated display video,
The one or more cameras are arranged in one of the two regions partitioned by the wall,
The display unit or a screen for projection by the display unit is attached to the wall in the other region of the two regions partitioned by the wall,
The display video generation unit is configured so that the position of the face detected by the detection unit is located on the left side of the center of the screen for projection by the display unit or the display unit. Generating a display image including a surrounding image on the right side of the display unit or a screen for projection by the display unit, and the position of the face detected by the detection unit is The closer to the right side of the center of the screen for the projection by the display unit or the display unit, the more the left side of the generated surrounding image toward the screen for the projection by the display unit or the display unit. Generate a display image that includes a certain surrounding image ,
The display video generation unit
A conversion map creating unit that creates a conversion map that defines pixels of the surrounding video corresponding to each pixel of the display video;
A conversion map memory for storing the created conversion map;
A conversion unit that generates the display video by rearranging the pixels in the surrounding video memory according to the conversion map stored in the conversion map memory;
The conversion map creating unit specifies a part of the surrounding video, calculates a primary conversion formula for correcting the part of the specified surrounding video to fit the size of the display area of the display unit, A monitoring device that creates the conversion map based on the primary conversion formula . A monitoring device that monitors one of two regions partitioned by a wall,
One or more cameras for photographing the surroundings;
A surrounding image generation unit that generates a wide range of surrounding images by shooting with the camera;
A peripheral video memory for storing the peripheral video generated by the peripheral video generator;
A detection unit for detecting the position of the user's face;
In accordance with the position of the face detected by the detection unit, a part of the surrounding video stored in the surrounding video memory is specified, and a display video that generates the display video including the specified part is generated. A video generator;
A display unit for displaying the generated display video,
The one or more cameras are arranged in one of the two regions partitioned by the wall,
The display unit or a screen for projection by the display unit is attached to the wall in the other region of the two regions partitioned by the wall,
The detection unit detects a distance between the display unit or the screen for projection by the display unit and the face unit,
The display image generation unit generates a display image that includes a wider range of images in the generated surrounding image as the distance detected by the detection unit is shorter,
The display video generation unit
A conversion map creating unit that creates a conversion map that defines pixels of the surrounding video corresponding to each pixel of the display video;
A conversion map memory for storing the created conversion map;
A conversion unit that generates the display video by rearranging the pixels in the surrounding video memory according to the conversion map stored in the conversion map memory;
The conversion map creating unit specifies a part of the surrounding video, calculates a primary conversion formula for correcting the part of the specified surrounding video to fit the size of the display area of the display unit, A monitoring device that creates the conversion map based on the primary conversion formula .   The monitoring device according to claim 5 or 6, wherein the face detected by the detection unit is at least one of the user's face, eyes, nose, eyebrows, and mouth. The detection unit detects a position between the user's eyebrows,
The display image generation unit includes a vertical line drawn from a position between the eyebrows to a screen for projection by the display unit or the display unit, and the vertical line when viewed from above, in the generated surrounding image. Includes a left surrounding image in the range of angles sandwiched by a boundary line rotated counterclockwise by an angle α0, and rotates the perpendicular line and the perpendicular line by an angle α1 when viewed from above. Display video that includes the right surrounding video in the range of angles between the border
The distance from the point where the perpendicular intersects the display unit or the screen for projection by the display unit to the left end of the screen for projection by the display unit or the display unit is b0, the distance to the right end is b1, When the length of the perpendicular is a,
The monitoring apparatus according to claim 1, wherein α0 = tan ⁻¹ (b0 / a) and α1 = tan ⁻¹ (b1 / a).

Images