WO2019153855A1

WO2019153855A1 - Object information acquisition system capable of 360-degree panoramic orientation and position sensing, and application thereof

Info

Publication number: WO2019153855A1
Application number: PCT/CN2018/118924
Authority: WO
Inventors: 罗镇邦
Original assignee: 迎刃而解有限公司
Priority date: 2018-02-07
Filing date: 2018-12-03
Publication date: 2019-08-15
Also published as: TWM580186U

Abstract

Provided is an object information acquisition system capable of 360-degree panoramic orientation and position sensing on the basis of digital 360-degree fisheye image capturing. The object information acquisition system capable of 360-degree panoramic orientation and position sensing comprises a fisheye image capturing unit, an object detection unit, an object information acquisition unit, an object orientation computation unit, and a system positioning computation unit. The object information acquisition system capable of 360-degree panoramic orientation and position sensing acquires, by means of local image identification or scanning and the Internet, object information enabling identification of an object. Further provided are various applications based on the object information acquisition system capable of 360-degree panoramic orientation and position sensing.

Description

360 degree surround orientation and position sensing object information acquisition system and application thereof

Technical field

The present invention relates to the field of optical positioning and intelligent information acquisition and processing, and in particular to a 360 degree surround orientation and position sensing object information acquiring system and application thereof.

Background technique

The traditional mobile positioning methods are mainly as follows:

1, GPS positioning method

Conventional GPS positioning uses electromagnetic wave signals, and its positioning accuracy is often affected by factors such as building blockage, indoor or underground shielding, and multiple channels formed by reflection of buildings or walls. Therefore, it is only suitable for outdoor positioning, and its positioning accuracy is measured in meters (meters). In urban environments where buildings are mostly 10 meters or less, users often get inaccurate maps by inaccurate GPS positions. The location is misleading.

2, WI-FI positioning

In general, WI-FI positioning has higher accuracy than cellular network positioning. It uses the wireless access point and measures the strength of the signal received from one or more network forming ends to determine the position of the object under test. However, WI-FI positioning is very expensive and it can only achieve a positioning accuracy of 5-10 m when it has a large number/high density of WI-FI access points.

3. Other positioning methods

Ordinary laser/ultrasonic range finder also has applications in ranging, but it is only used for ranging and does not provide information on the position and direction of the object to be measured and the laser itself has a harmful effect on the human eye. Therefore, conventional laser and ultrasonic ranging do not provide 360-degree surround position and direction information.

With the advancement of mobile communication networks, radio frequency identification technology is rapidly developing. The low-precision positioning that relies on electromagnetic wave conduction signals in the prior art has been unable to meet the requirements for precise indoor and outdoor positioning. For example, in the field of blind navigation, when blind people walk indoors, traditional navigation is often limited, and its positioning within a few meters often fails. Moreover, even with Wi-Fi positioning to improve positioning accuracy, positioning within a few meters needs to rely on very accurate indoor maps for support, which requires the associated map software to have strong data support, and it needs timely updates to avoid buildings. Or the positioning caused by the change of the store name is invalid.

Image, as a real-time effective carrier for carrying position information, has unique advantages in real-time collection of environmental information. It does not have electromagnetic signal interference, positioning accuracy and other aspects, and the image can be collected in real time. However, the existing image recognition technology is far from being able to accurately recognize the orientation information of the positioning object only by the single-shot picture. In the existing radio frequency identification technology, the two-dimensional code has been obtained by scanning, so that the network URL is associated with the application of the corresponding payment function, but the image recognition and scanning code technology is not applied to the 360-degree surrounding position and orientation. Use. Moreover, with the development of science and technology, relying on precise positioning to achieve the use of various functions more and more. For example, in the field of blind people and robot navigation, the positioning accuracy needs to be higher to achieve its function, and the existing several positioning methods have a large gap between the accuracy and the actual demand.

On the other hand, RF-based (eg Wi-Fi) positioning technology, especially in the case of robotics and commercial applications, for example, in a public environment, can transmit RF signals through their devices to interfere with RF positioning signals, thereby Destroy or attack a hidden intruder (such as in a bathroom in a mall). This problem of hidden intruders can be avoided. The present invention provides a non-RF positioning method in which an intruder must block the line of sight between the device and the observed object, and the behavior of the intruder can be recorded by the camera of the device or closed in the environment. TV security monitoring.

technical problem

The present invention is directed to solving at least some of the above technical problems or at least providing a useful commercial application. To this end, the first object of the present invention is to provide a 360-degree surround orientation and position sensing object information acquisition system based on 360-degree digitally photographed fisheye images (A 360-degree surround direction and position aware object information retrieval System based on 360-degree fisheye digital camera images), the 360-degree surrounding azimuth and position sensing object information acquiring system includes a fisheye image capturing unit, an object detecting unit, an object orientation computing unit, a system positioning operation unit, and an object information acquiring unit, The fisheye image capturing unit is configured to capture the surrounding environment to be recognized to obtain an overall fisheye image of the environment to be identified; the object detecting unit is configured to detect at least one identifiable object in the environment to be identified; object information The obtaining unit obtains information about the object from the network (for example, the Internet), such as the global location of the object, the environment map and location of the object, or other information; the object orientation computing unit is configured to calculate each identifiable object according to the imaging parameters of the fisheye camera. a 360-degree orientation relative to the fisheye camera; the system positioning operation unit is configured to calculate based on the triangulation method according to the orientation information of the at least two different identifiable objects provided by the object orientation operation unit with respect to the fisheye camera 360 degree spatial distance and position of the object relative to the system And information acquisition unit from an object or is embedded in the object (for example, two-dimensional code is embedded in the label) object or environment of global map position to achieve absolute positioning operation of the system of global environment or location. A second object of the present invention is to provide an intelligent telematics communication system, and more particularly to an intelligent remote shopping system using a 360 degree surround orientation and position sensing object information acquisition system. A third object of the present invention is to provide an intelligent car navigation system. A fourth object of the present invention is to provide an intelligent guide blind. Furthermore, high-density, large-scale RF devices currently used by many indoor positioning systems, such as Wi-Fi access points, may also have an impact on people's physical health. The invention significantly avoids the harm to human physiological health through the optical positioning method.

Technical solution

A 360-degree circumferential orientation and position sensing object information acquiring system based on a 360-degree digital fisheye image of the present invention, and a 360-degree fisheye image obtained by each fisheye observation point of the fisheye image capturing unit is The image processing method performs a 360-degree surrounding object detection on the overall environment surrounding the fisheye observation point of the system to sense the 360-degree orientation of the object and the 360-degree spatial distance and position of the perceived object, 360-degree circumferential orientation and position sensing object information. The acquisition system includes:

The fisheye image capturing unit, the fisheye image capturing unit uses at least one fisheye digital camera to photograph the surrounding environment to be recognized, and the optical axis of the fisheye digital camera is perpendicular to the viewing plane (plane) Of direction of view, or view plane), to obtain a 360-degree surrounding overall fisheye image of the environment to be identified;

The object detecting unit receives the output image of the fisheye image capturing unit, and detects at least one identifiable object from the overall fisheye image according to different object categories;

The object information acquiring unit, the object information acquiring unit is configured to: for the identifiable object, if the object type is a label identifiable object, the image of the label object is partially scanned/identified to extract the included object information, including but not limited to the object information. The object's network (such as the Internet) information address, and real-time access to the object network (such as the Internet) information address to obtain the object's network (such as the Internet) information, if the object category is a non-label type identifiable object, then the object image in real time Identifying objects and obtaining identifying object information through a network (eg, the Internet) image recognition service;

The object orientation calculation unit receives the detection result of the object detection unit, and is used to determine the correspondence of each identifiable object in the overall fisheye image for each identifiable object detected by the object detection unit. Positioning points and calculating a 360 degree orientation of each identifiable object relative to the fisheye camera in the environment to be identified based on the imaging parameters of the fisheye camera;

a system positioning operation unit, the system positioning operation unit receives the operation result of the object orientation operation unit, and is configured to calculate the direction information of the at least two different identifiable objects provided by the object orientation operation unit relative to the fisheye camera, based on the triangulation method The 360-degree spatial distance and position of the object relative to the system, and the global or environmental map of the identifiable object obtained from the object information acquisition unit or the information contained in the object (eg, included in the QR code label) Absolute position, the exact location of the system's global or environmental map is calculated from the objects in the surrounding observation environment that provide location information.

Preferably, the tag-like identifiable object is a specially customized identifiable tag pattern; the object information acquiring unit obtains a network (eg, Internet) channel bound to the identifiable object by locally scanning the object code on the identifiable tag pattern, and passes Access network (eg internet) channels to get object information.

Preferably, the non-label type identifiable object is a specific object contour pattern preset by the system, and the object detecting unit performs object detection of the entire image by an artificial intelligence image service built in or through a network (for example, the Internet).

Preferably, the identifiable label pattern is a high contrast double square pattern, and the object code is a two-dimensional code for obtaining a paste of an object network (eg, the Internet) channel or printed in a double square pattern.

Preferably, a Morse Code encoding the basic information of the identifiable object is overlaid on the double square pattern.

Preferably, the method further comprises:

The optical telescope unit, the optical telescope unit is connected to the system positioning operation unit for digital shooting, and is used for automatically aiming and/or focusing and digitally capturing the identifiable object according to the relative positioning information provided by the system positioning operation unit.

Preferably, the method further comprises:

An object information output unit, the object information output unit is connected to the system positioning operation unit and/or the object information acquisition unit, and the object information output unit comprises a display unit and/or a voice unit;

The display unit is configured to visually display at least one of direction information, positioning information, and object information of the identifiable object; the voice unit is configured to audibly broadcast at least one of direction information, positioning information, and object information of the identifiable object Kind.

Preferably, in the fisheye image capturing unit, the number of fisheye cameras is one, which is configured to perform photographing at at least two different positions, so that the object orientation computing unit can provide at least two different sets of cameras relative to the fisheye. Direction information.

Preferably, in the fisheye image capturing unit, the number of fisheye cameras is two or more positions that are relatively fixed in position, so that the object orientation arithmetic unit can provide at least two sets of different orientation information with respect to the fisheye camera.

Preferably, the object information acquisition system further comprises a remote output unit, the remote output unit comprises a remote panoramic augmented reality device, and the panoramic augmented reality device comprises a reflective surround display device, the overall environment to be recognized received from the object information acquisition system The fisheye image is displayed on the reflective surround display device in a panoramic scene, wherein the object information of the identifiable object detected by the object information acquisition system is displayed in an augmented reality in the panoramic scene.

The invention also provides an intelligent telematics communication system, comprising the above 360 degree surround orientation and position sensing object information acquiring system.

The invention also provides an intelligent car navigation device, which can perform positioning and intelligent navigation according to identifiable objects in the surrounding environment, and is equipped with the above-mentioned 360-degree surrounding azimuth and position sensing object information acquiring system.

Preferably, a magnetometer device is also included, the magnetometer device being operative to cooperate with the system positioning arithmetic unit to provide absolute positioning information of the smart car navigation system.

The invention also provides an intelligent guide blind 仗, which can provide guidance information for the visually impaired person to identify the identifiable object in the surrounding environment, and install the above-mentioned 360-degree surrounding azimuth and position sensing object information acquiring system, and the guiding information includes identifiable information. At least one of direction information, positioning information, and object information of the object.

Preferably, a magnetometer device is also included, the magnetometer device being operative to cooperate with the system positioning arithmetic unit to provide absolute positioning information of the intelligent guide blind.

Beneficial effect

Using the fisheye lens to take photos in the environment can overcome many problems with WI-FI and GPS positioning, because the fisheye lens is an optical principle, which is different from electromagnetic wave signal transmission, and there will be signal loss when electromagnetic waves are transmitted in space. Buildings block, because reflection causes multiple channels and other problems; and pictures as a good carrier of position information, there is no such problem, and can be achieved by taking pictures in the environment, at close range or even close distance Positioning has a unique advantage. However, in the prior art, relying solely on the image recognition technology, it is quite technically difficult to accurately obtain the geographical position of the identifiable object from the single or multiple captured images; however, by recognizing the captured fisheye image Object code is no longer a technical problem. Therefore, the fisheye image is captured, and the graphic recognition object code can accurately locate the geographical position of the identifiable object through the communication connection network end, which is different from directly transmitting the electromagnetic wave signal through the mobile terminal to determine the geographical position of the identifiable object, and pass the object in advance. The code will record the geographical location of the identifiable object to avoid the signal loss of the electromagnetic wave signal during transmission, and the uncontrollable failure positioning caused by the multi-path after the building block or reflection, avoiding each positioning process (for the same positioning target) because The positioning error caused by the difference in real-time signals, thus misleading the user to the target location that is not expected to arrive.

Secondly, by setting additional technical features such as a cloud-top optical telescope with digital shooting, the invention can improve the recognition and positioning function for long distances and even small objects; and, by the precise positioning of the present application, the prior art can be realized. Because of the lack of positioning accuracy, many practical applications, such as autonomous robots or blind people, are difficult or difficult to achieve when they go to unfamiliar public places.

The additional aspects and advantages of the invention will be set forth in part in the description which follows.

DRAWINGS

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from

1 is a schematic diagram showing the cooperation principle between a core functional unit, a core data element, and a peripheral device of a 360 degree surround orientation and position sensing object information acquiring system according to a preferred embodiment of the present invention;

2 is a structural block diagram of a 360-degree surround orientation and position sensing object information acquiring system according to an embodiment of the present invention;

3 is a schematic diagram showing the calculation of the relative direction calculation of the identifiable object and the fisheye lens in the fisheye image when shooting with a fisheye lens according to an embodiment of the present invention;

4A is a cross-sectional view of a 360-degree direction (in an inverted dome type) taken in a direction perpendicular to the viewing plane of the fisheye lens with a vertical FOV greater than 180 degrees, in accordance with an embodiment of the present invention. Cross-sectional schematic view;

4B is a schematic diagram showing the relationship between the elevation angle and the image resolution when the optical axis direction of the fisheye lens is vertically downward with respect to the viewing plane;

4C is a schematic diagram showing the relationship between the elevation angle and the image resolution when the optical axis of the fisheye lens is vertically upward with respect to the viewing plane;

5 is a structural block diagram of a 360 degree surround orientation and position sensing object information acquiring system according to another embodiment of the present invention;

6 is a structural block diagram of a 360-degree surround azimuth and position sensing object information acquiring system according to still another embodiment of the present invention;

7 is a schematic diagram of a 360 degree surround orientation and position sensing object information acquiring system applied to a smart car navigation system according to an embodiment of the present invention;

8 is a schematic diagram of a 360 degree surround orientation and position sensing object information acquiring system applied to an intelligent guide blind 仗 according to an embodiment of the present invention;

Figure 9 is a schematic view showing an embodiment of a different installation form of a fisheye camera and a modification thereof in the embodiment of the present invention;

10 is a schematic diagram of a 360 degree surround orientation and position sensing object information acquiring system applied to an intelligent remote shopping system according to an embodiment of the present invention;

11 is a schematic diagram of a robot with a fisheye camera incorporating a 360 degree surround orientation and position sensing object information acquisition system according to an embodiment of the present invention, respectively showing a fisheye camera and three fisheyes. Camera robot

Figure 12 is a schematic illustration of an easily identifiable label pattern in an embodiment of the present invention showing a two-dimensional code disposed at the center of the label pattern and a Morse code disposed at the edge of the label pattern.

Embodiments of the invention

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. To limit the invention.

The invention provides a 360 degree surround orientation and position sensing object information acquiring system based on 360 degree digital fisheye image (A 360-degree surround direction and Position aware object information retrieval system based on 360-degree fisheye Digital camera images), a 360-degree fisheye image obtained by each fisheye observation point of the fisheye image capturing unit, and an image processing method for 360-degree surrounding object detection of the overall environment surrounding the fisheye observation point of the system To sense the 360-degree orientation of the object and the 360-degree spatial distance and position of the perceived object, wherein the label-identifiable object identifies the image of the label-recognizable object in the overall image by partial scanning to extract the embedded object The information includes, but is not limited to, a network (e.g., Internet) information address, and accesses the object network (e.g., the Internet) information address in real time to obtain a network (e.g., Internet) information of the object, and the object image is obtained for the non-label type identifiable object. An image recognition service via a network (such as the Internet) to identify the object and obtain object information.

1 is a schematic diagram showing the principle of cooperation between a core functional unit of a 360-degree surround orientation and position-aware object information acquisition system based on a 360-degree digitally imaged fisheye image and a peripheral optional device and a core data element according to a preferred embodiment of the present invention; 2 is a block diagram showing the core structure of a 360-degree surround orientation and position-aware object information acquisition system based on a 360-degree digitally imaged fisheye image according to an embodiment of the present invention. As shown in FIG. 1, the 360 degree surround orientation and position sensing object information acquisition system implements various functions and applications of the present invention by working in conjunction with peripheral optional devices by its core functional unit, wherein the left side of the figure is listed. The external input or optional device required to complete the work of the core function unit of the object information acquisition system (starting the trigger of the fisheye camera, the object type information input by the user, the system working range set by the user, the cloud table with the digital camera Optical telescopes, magnetometer devices, etc.), the core data elements generated from each core functional unit are listed on the right side of the figure, and the final output information of the 360-degree surround orientation and position-aware object information acquisition system is shown below (eg The video or audio, etc., with respect to the core functional units of the present invention will be explained in further detail below with reference to FIG. 2.

Referring to FIG. 2, the present invention provides a 360 degree surround orientation and position sensing object information acquiring system. The object information acquiring system includes a fisheye image capturing unit 10, an object detecting unit 20, an object information acquiring unit 30, and an object orientation computing unit. 40. The system positioning operation unit 50 respectively corresponds to each core function unit shown in FIG. 1, and the functions thereof are respectively briefly described below.

The fisheye image capturing unit 10 photographs the surrounding environment to be recognized using a fisheye camera to obtain an overall fisheye image of the environment to be recognized. Specifically, the fisheye image capturing unit 10 of the embodiment of the present invention uses one or more fisheye cameras to photograph the surrounding environment to be recognized, and the optical axis of the fisheye digital camera is perpendicular to the plane of view (plane of Direction of view, or view plane), to obtain a 360-degree overall fisheye image of the environment to be identified. Since the fisheye camera has the characteristics of short focal length and large viewing angle, the overall fisheye image captured by the fisheye image is larger. Image; moreover, the use of ultra-wide-angle fisheye camera for shooting in the identification environment can overcome the traditional limitations of WI-FI and GPS positioning, because the image captured by the fisheye camera is generated based on the optical principle, which is different from the electromagnetic wave signal. Conduction, there are signal losses when electromagnetic waves are transmitted in space, or positioning failure due to building blockage; on the contrary, images are unique as location information, especially in indoor environments.

The object detecting unit 20 communicates with the fisheye image capturing unit 10 for detecting at least one identifiable object located in the environment to be recognized from the overall fisheye image according to different object categories. Specifically, the object detecting unit 20 receives the overall fisheye image captured by the fisheye image capturing unit 10, and then according to different object categories (the object category can be classified into a tag type identifiable object and a non-label type identifiable object) from the whole. At least one identifiable object located in the environment to be identified is detected in the fisheye image, thereby ultimately achieving optically accurate positioning of the identifiable object. The object category is used to define a specific pattern or contour that the object detecting unit 20 detects from the overall fisheye image, and the user can arbitrarily set or select the object category according to different applications and actual situations. In a specific embodiment, the object category may be a specially customized label-like identifiable object, such as an easily identifiable label pattern, or a system-preset non-label identifiable object, such as a specific object outline pattern (person, table, Cars, etc.).

For the identifiable object, if the object category is a tag-type identifiable object, the image of the tag object in the overall image is scanned/identified to extract the included object information, including but not limited to the network of the object ( For example, the Internet) information address, and real-time access to the object network (such as the Internet) information address to obtain the network (such as the Internet) information of the object, if the object category is a non-label type identifiable object, the object image is transmitted through the network (for example Internet) image recognition service to identify objects and obtain identification object information;

The object orientation computing unit 40 communicates with the object detecting unit 20 for determining, for each identifiable object detected by the object detecting unit 20, a corresponding position point of each identifiable object in the overall fisheye image, and A 360 degree orientation of each identifiable object relative to the fisheye camera in the environment to be identified is calculated from the imaging parameters of the fisheye camera. A detailed procedure on how to perform the above-described direction information calculation using the fisheye image will be further explained below with reference to FIG.

The system positioning operation unit 50 communicates with the object orientation operation unit 40 and/or the object information acquisition unit 30 for using the orientation information of at least two different sets of identifiable objects provided by the object orientation operation unit 40 with respect to the fisheye camera. Calculating the 360-degree spatial distance and position of the object relative to the system based on triangulation, and obtaining globally identifiable objects from the object information acquisition unit or the information contained in the object (eg, included in the QR code label) Or the absolute location of the environment map, the exact location of the system's global or local environment map is calculated from the objects in the surrounding observation environment that provide location information. Incidentally, the triangulation method is a well-known method in the field of positioning technology, and is widely applied in the conventional positioning method, and therefore this article will be briefly explained.

Specifically, in the object information acquiring system of the embodiment of the present invention, when the object category is an easily identifiable label pattern, the object information acquiring unit 30 scans the object code on the easily identifiable label pattern to obtain a network bound to the identifiable object (for example, the Internet). Channel, and then access any network object information by accessing the network (such as the Internet) channel, which is simple and fast, and can obtain object information (such as audio, video or text) in multimedia form and is not easy to make mistakes. When the object category is a specific object contour pattern preset by the system, the object detecting unit performs the overall image by an artificial intelligence image service (such as face recognition technology, object contour recognition technology) built in or through a network (for example, the Internet). Object detection, which enhances the user experience.

In a specific implementation, the easily identifiable label pattern is disposed at an appropriate position in the environment to be recognized, so that the fisheye image capturing unit 10 performs photographing. Generally, the easy-to-identify label pattern is set at a position suitable for the fish-eye image capturing unit 10 to be photographed in the environment to be recognized, so that the fish-eye image capturing unit 10 can photograph the easily-recognizable label pattern, thereby further facilitating the object information. The obtaining unit 30 scans the object code on the identifiable label pattern to obtain a network (for example, Internet) URL bound to the identifiable object, and obtains object information by accessing a network (for example, the Internet) website, thereby improving the recognition rate and improving the recognition rate. Identify efficiency.

In a specific implementation, the easily identifiable label pattern is preferably a high contrast double square pattern, in which case the item code can be a two dimensional code affixed or printed within the double square pattern. Referring to FIG. 12, the image on the left side of FIG. 12, the easy-to-identify label pattern is a high-contrast double-square pattern, and the object code is a two-dimensional code pasted or printed in a double square pattern, and the object information acquisition system of the embodiment of the present invention passes The object information acquiring unit 30 scans the two-dimensional code to obtain a network (for example, Internet) channel bound to the identifiable object, and obtains the object information by accessing a network (for example, the Internet) channel, and the object information acquiring system of the embodiment of the present invention passes Scanning the QR code to obtain the object information has the advantages of large amount of information, easy identification, and low cost. In a specific implementation, the Morse code containing the basic information of the identifiable object can be further embedded in the edge of the double square pattern. Referring to FIG. 12, the image located on the right side of FIG. 12, Morse Code is embedded in the edge of the double square pattern. The example Morse code in the figure means a road sign, and the Morse code can also include an identifiable object. Map location information, other basic information, etc. The object information acquiring system of the embodiment of the present invention expands the specific use range of the easily identifiable label pattern by embedding the Moir code of the basic information contained in the identifiable object in the edge of the double square pattern, and the use field is wide.

It should be noted that, in the object information acquiring system of the embodiment of the present invention, when the object detecting unit 20 detects the identifiable object from the overall fisheye image, the object category is not particularly limited as long as the identifiable object It can be easily identified by existing image recognition technology. In fact, depending on the application and business model, the identifiable items can be maintained and set up by specific participants, such as in a mall, which can be set by the property company, and can be pre-identified by the manufacturer on a particular product.

As one of the use examples, the object information acquisition system further includes a remote output unit, the remote output unit includes a remote panoramic augmented reality device, and the panoramic augmented reality device includes a reflective surround display device, which is received from the object information acquisition system. The overall fisheye image of the recognition environment is displayed on the reflective surround display device in a panoramic scene, wherein the object information of the identifiable object detected by the object information acquisition system is displayed in a panoramic manner in an augmented reality manner. The process of obtaining a fisheye image using a fisheye camera and calculating the direction information of the identifiable object relative to the fisheye camera in an actual shooting environment is described below with reference to FIGS. 3-4, for the same identifiable object, according to Two or more directions information is used to calculate the position information.

3 is a schematic diagram showing the principle of calculating relative direction information of an identifiable object and a fisheye lens in a fisheye image when photographing with a fisheye lens according to an embodiment of the present invention; FIG. 3 is shown below and The principle of object orientation (elevation and azimuth with respect to the viewing plane) resulting from the principles described in 4 (4A, 4B, and 4C). In the figure, using the optical axis pointing downward with respect to the viewing plane, the image position P (for example, the center of the detected object or a reference point on the object) is detected for the detected object in the object detecting unit. Point P' (ie the position of point P in the picture). The actual corresponding position of point P' in space is the spatial position point P, which has an elevation angle with respect to the viewing plane and an azimuth with respect to the system direction reference line.

The point P' is spaced from the optical axis of the fisheye image by an axial distance R (in pixels), and the corresponding P point of the point P with respect to the viewing plane is calculated by the fisheye function of the fisheye using the R value. The angle with the optical axis (in this case, the nadir angle). Obviously:

Elevation angle = angle between P point and optical axis – 90 degrees

Obviously, the azimuth of the direction reference line relative to the system is the angle formed by the radial line of the point P' on the fisheye image with the reference line.

4A is a schematic side view showing the orientation of the optical axis of the fisheye lens perpendicular to the viewing plane; and FIG. 4B is a view showing the relationship between the elevation angle and the image resolution when the optical axis of the fisheye lens is vertically downward with respect to the viewing plane. FIG. 4C is a schematic diagram showing the relationship between the elevation angle and the image resolution when the optical axis of the fisheye lens is vertically upward with respect to the viewing plane.

4A depicts a cross-sectional view of the orientation of the fisheye optical axis relative to the viewing plane in the fisheye image capture unit. Referring to the drawings, the viewing plane defines the zero elevation angle of the three-dimensional view field at the fisheye viewing point of the 360 degree surround view, and includes the line of sight. The viewing direction may be any viewing direction, or it may be the front of the system or the forward/moving direction, as in the case of the "smart positioning driving" application of the invention described in the following patent specification; the viewing direction may be a non-ground plane Direction, such as the direction of the robot's eye. Likewise, the viewing plane may be a ground plane or a non-ground plane that varies with the orientation of the system, as in the case of the smart guide stick application described below in this patent specification. In the present invention, the orientation of the fisheye optical axis should be oriented in a direction perpendicular to the viewing plane of the viewing point (e.g., by fixture or mounting), i.e., pointing upward or downward relative to the viewing plane to obtain an environment to be identified. The 360 degree surrounds the system's overall fisheye image. The space of all visible directions of the fisheye can be inverted with a dome (as shown in Figure 4A, the fisheye axis is vertically pointing downward with respect to the viewing plane), and the non-inverted dome (when the fisheye axis is pointing vertically with respect to the viewing plane) Description, as described by the dotted line in the cross-sectional view. Obviously, in the reference figure, the larger the FOV of the fisheye/super fisheye, the more it can observe and detect the spatial orientation of the objects used in the surrounding environment using the 360-degree fisheye image.

A different perspective view of the fisheye camera is shown in Figures 4B and 4C, with the optical axis pointing downward in Figure 4B and pointing upward in Figure 4C. For the described fisheye optical axis orientation, the same elevation angle (or elevation angle) is located on a circle in the resulting fisheye image with respect to the equal elevation line of the viewing plane. All directions of the viewing plane (ie, 360 degrees) are located on a circle representing the zero elevation angle (ie, an elevation line such as zero degrees). Similarly, all 360 degree directions of any other elevation angle are also represented by their corresponding circles around the center of the fisheye optical axis. As a result, the fisheye image in Figures 4B and 4C represents a 360 degree surround image with a viewing angle of zero elevation.

In FIG. 4B in which the optical axis is directed downward with respect to the viewing plane, the elevation angle changes from a negative value to a zero degree (viewing plane) to a positive value when moving outward from the center of the circular fisheye image. As we move outward, the circumference of the circle and the number of pixels increase, so we see that the "circular" resolution of the image is also higher for higher elevation angles (or elevation angles) relative to the viewing plane. The corresponding converted flat (rectangular) panorama is displayed on the right side, with the lower part being the lower elevation angle and the upper part being the higher elevation angle. If a circular fisheye image is converted to this standard flat image, the upper portion will have a higher resolution for a higher elevation angle. Since the "circular" resolution (ie, the number of pixels) of the lower elevation angle in the initial circular fisheye image is lower than the "circular" resolution (ie, the number of pixels) of the higher elevation angle, the lower panoramic image is lower. The elevation angle actually has a lower resolution, so different degrees of image magnification, such as image interpolation, are required to achieve the same number of pixels as in the higher elevation angle. As a result, in this case (where the fisheye optical axis points downward with respect to the viewing plane), a higher elevation angle is better for object detection and image processing accuracy/accuracy. In Figure 4C, where the optical axis is pointing relative to the viewing plane, and vice versa, the lower elevation angle has a higher "circular" resolution than the higher elevation angle. As a result, in this case (where the fisheye optical axis points upward with respect to the viewing plane), the lower elevation angle is better for object detection and image processing accuracy/accuracy. With high-resolution light-sensing components that currently provide very high-resolution digital images, high image processing accuracy/accuracy can be achieved at elevations with less "circular" resolution/pixels.

Referring to FIG. 5, on the basis of the preferred embodiment shown in FIG. 2, the object information acquiring system according to another embodiment of the present invention further includes an optical telescope unit 60, and the optical telescope unit 60 is connected to the system positioning operation unit 50. The aiming and focusing of the identifiable object are automatically performed according to the relative positioning information provided by the system positioning operation unit 50. The object information acquiring system of the embodiment of the present invention can automatically perform the aiming and focusing of the identifiable object and the digital shooting by setting the optical telescope unit 60, thereby overcoming the recognition caused by the distance between the identifiable object and the small object. The problem of lower degree is higher, that is, the object information acquiring system of the embodiment of the present invention has a unique advantage by means of the optical telescope unit 60 in both long-distance and close-range positioning.

Further, referring to FIG. 6, the object information acquiring system according to still another embodiment of the present invention further includes an object information output unit 70, and the object information output unit 70 is connected to the system positioning operation unit 50 and/or the object information acquiring unit 30, and the object information is output. The unit 70 is configured to output object information, object direction information, and object orientation information.

The object information output unit 70 further includes a display unit and/or a voice unit, that is, the object information output unit 70 may include one of a display unit and a voice unit, and may also include a display unit and a voice unit. The display unit is configured to visually display at least one of direction information, positioning information, and object information of the identifiable object; the voice unit is configured to audibly broadcast at least one of direction information, positioning information, and object information of the identifiable object Kind.

In a specific implementation, in the fisheye image capturing unit 10, the number of fisheye cameras may be one, and shooting is performed at at least two different positions to obtain a fisheye image, so that the object orientation computing unit 40 can provide at least two different sets. Directional information relative to the fisheye camera. By setting the number of fisheye cameras to one and arranging for shooting at at least two different locations, not only can at least two different sets of directional information relative to the fisheye camera be provided, but also the number of fisheye cameras Only one can reduce costs and shrink size.

In a specific embodiment, the number of fisheye cameras of the fisheye image capturing unit 10 is preferably two or more positions that are relatively fixed in position, so that the object orientation computing unit 40 can provide at least two different sets of cameras relative to the fisheye. Direction information. By arranging the number of fisheye cameras to two or more positions that are relatively fixed, not only can the function of providing at least two different sets of directional information relative to the fisheye camera be achieved, but also due to the two or more fisheye cameras The number is relatively fixed, which avoids moving or adjusting the camera position, which saves time and improves work efficiency.

Hereinafter, several specific application examples of the 360-degree surround orientation and position sensing object information acquiring system according to the above embodiment of the present invention will be described with reference to FIGS. 7-11.

Referring to FIG. 7, the present invention provides an intelligent car navigation system capable of positioning and intelligently navigating according to an identifiable object in a surrounding driving environment, and the smart car navigator is equipped with the above-mentioned 360-degree digital shooting fisheye image. 360 degree surround orientation and position sensing object information acquisition system. Preferably, a high-contrast double square pattern is set as an easily recognizable label pattern on a street sign or a building on both sides of the road, and a two-dimensional code pattern is printed or pasted in the center of the double square pattern, by scanning the two-dimensional code and accessing the network (for example, the Internet) can get any information about the location of the car or the surrounding buildings. A magnetometer device can be provided, the magnetometer device being used in conjunction with the system positioning arithmetic unit to provide absolute positioning information of the smart car navigation system.

Referring to FIG. 7, the smart car navigation system of the embodiment of the present invention has the following steps for positioning and intelligently navigating according to identifiable objects in the surrounding environment:

The driver activates a smart car navigator that includes a 360 degree surround orientation and position sensing object information acquisition system based on a 360 degree digital fisheye image. The smart car navigator can be integrated into the vehicle or installed as a stand-alone device in a suitable location on the vehicle. Preferably, the smart car navigation system comprises two fisheye cameras, which are respectively disposed on the left and right sides of the vehicle near the front windshield, and the driver activates the smart car navigation system including the 360 degree surrounding orientation and the position sensing object information acquiring system. After that, the fisheye camera of the smart car navigator began to capture images of the surrounding environment.

The fisheye image capturing unit including two fisheye cameras captures a 360° surrounding environment image; the object detecting unit detects a double square pattern in the captured image as an easy-to-identify label pattern; and the object information acquiring unit scans the double square figure The two-dimensional code in the case is linked to a specific web address to read the position information of the high-precision global map; the object orientation operation unit calculates the direction information of the double square pattern; the system positioning operation unit uses the triangulation method to measure the relative of each double square pattern. location information.

Specifically, after the smart car navigation device starts working, the fisheye image capturing unit starts capturing and capturing the surrounding environment to be recognized, usually a 360° surrounding environment, in order to obtain an overall fisheye image of the environment to be identified. The object detecting unit obtains a double square pattern from the overall fisheye image, and the double square pattern includes two-dimensional code information. The object information acquiring unit reads the position information of the high-precision global map by using the link URL provided by the two-dimensional code in the double square pattern, and the map position information usually includes the longitude and the latitude, and the high precision can be obtained by the unique determination of the longitude and the latitude. Location information for the global map. The object orientation arithmetic unit calculates direction information from the object to be identified (for example, the double square pattern). The system positioning operation unit uses triangulation to measure the relative position information of each object to be identified relative to the smart car navigation system. In addition, the detection flag for setting the double square pattern may include a road sign, a store sign, and the like, and the advertisement information of the store includes audio information and video information that can be further downloaded from the network link URL of the logo.

The object information output unit can construct and output high-precision map position information based on the high-precision map position information obtained above. Optionally, the additional display unit of the system displays a brief description of the block map information of the vehicle, the network of the store logo (eg, the Internet) by means of the calculated map position and orientation; the driver may further click on the displayed location, Listen to the audio ad or watch the ad video to select the store location on the block map; the system display unit/speech unit outputs the audio ad or ad video of the selected store.

In a specific implementation, the detailed information about the identifiable object (for example, longitude information and latitude information) is input in advance, and the object information acquiring unit acquires, thereby avoiding each of the reasons caused by electromagnetic wave conduction in the real-time positioning process. Kind of error. First, since the relevant details of the identifiable object are actively input by the third party, the information is generally updated by the third party in real time, thereby avoiding the problem that the existing map is updated slowly and causing the positioning failure, for example, when a store is renamed, when used The shop's updated store name is used as the target arrival place. The existing maps often fail to locate the store name in the map. Secondly, the way to rely on electromagnetic wave signal positioning depends largely on receiving. The electromagnetic wave itself, and the user often has different environmental factors in each positioning process. For example, in a certain positioning, the signal emitted by the user is blocked by the obstacle, and the result is often the same positioning address input every time. But get different target locations. However, the above defects are solved by first obtaining the exact location of the identifiable object and then associating it with the map. Because the store is renamed, the latitude and longitude information of the store has not undergone any substantial change, so it will not be because of the map information. Not being updated in time or the name of the store has changed. The above examples are only used to explain the idea of the present invention. It is also understood that the present application is not limited to the case where the store is renamed, the building is demolished, the building is renamed, the merchant is renamed, or the like is not separated from the center. In this case, it should also be covered in the invention.

Referring to FIG. 8 and FIG. 9, the present invention further provides an intelligent guide blind, which can provide guidance information for an identifiable object in a surrounding environment to a visually impaired person, and the intelligent guide blind is installed with the above 360 degree surround. The orientation and position sensing object information acquiring system includes at least one of direction information, positioning information, and object information of the identifiable object.

Fig. 9 is a schematic view showing an embodiment of a different installation form of a fisheye camera and a modification thereof in the embodiment of the present invention. Taking the fisheye camera applied to the smart guide blind as an example, in the leftmost example of FIG. 9, the smart guide blind is provided with two fisheye image capturing units 10 in the vertical direction, wherein the fisheye located above The photographing direction of the image capturing unit 10 is upward, and the photographing direction of the fisheye image capturing unit 10 located below is downward, and the photographing ranges of the two fisheye image capturing units 10 overlap; in the example in the middle in FIG. The smart guide blind is provided with two fisheye image capturing units 10 in the vertical direction, wherein the fisheye image capturing unit 10 located above has a shooting direction upward, and the fisheye image capturing unit 10 located below is also photographed upward. The shooting ranges of the two fisheye image capturing units 10 also overlap; in the example of the rightmost side in FIG. 9, the smart guide blinds are provided with four fisheye image capturing units 10 in the vertical direction, and four fishes. The photographing directions of the eye image capturing unit 10 are all upward, and at least two of the fisheye image capturing units 10 have overlapping shooting ranges. In other embodiments, more fisheye image capturing units 10 can be disposed in the vertical direction, so that not only space in the horizontal direction can be saved, but also multiple fisheye image capturing units 10 can be used to obtain additional height measurements and Accurate measurement, for example, can be applied to higher robots or safety inspections, object search systems in high-rise buildings, and so on.

Referring specifically to FIG. 8, the visually impaired person walks on a blind road with a 360-degree surround orientation and position-sensing object information acquisition system based on a 360-degree digital fisheye image, and is provided alongside the blind road surface or the blind road. A double-square label pattern with a QR code, a store with drinks and food on both sides of the road (not necessarily with an easy-to-recognize label pattern), and a fisheye camera on the guide stick for 360 degrees of each system fisheye observation point Fisheye image, image processing method for 360 degree surrounding object detection of the overall environment surrounding the fisheye observation point of the system, to sense the 360 degree orientation of the object and the 360 degree spatial distance and position of the perceived object, wherein the label A class-recognizable object (a double-square label pattern with a QR code) identifies an image of the tag-recognizable object in the overall image by partial scanning to extract an information address of the embedded object network (eg, the Internet) as object information, and access The object network (such as the Internet) information address to obtain the network (such as the Internet) information of its object, while the non-label type identifiable object (Drinks, food, etc.) places the object image through the network (eg the Internet) image recognition service to identify the object and obtain property information. The direction information and the orientation information are calculated by the object orientation operation unit 40 and the system positioning operation unit 50, and the voice broadcast is performed. With the guide cane provided by the embodiment, the visually impaired person can accurately obtain the information of the beverage, the food, the street, and the like on the street by the object information acquiring unit 30, and obtain the specific object through the object orientation operation unit 40 and the system positioning operation unit 50. Directional information and orientation information, the guide cane is equivalent to the eyes of the visually impaired, convenient for the visually impaired. Can replace the guide dog, can also purchase items according to the prompts or find the services needed, the guide cans communicate with the network (such as the Internet), can also avoid the visually impaired, the family can monitor the travel of the visually impaired, may not Professionals and family members are the gospel of the visually impaired.

10 and FIG. 11, an intelligent telematics communication system according to an embodiment of the present invention is used for intelligent remote shopping, including the 360 degree surround orientation and position sensing object information acquiring system described above; and remotely interacting with the object information acquiring system. The panoramic augmented reality device, the panoramic augmented reality device comprises a reflective surround display device, and the overall fisheye image of the environment to be recognized received from the object information acquisition system is displayed on the reflective surround display device in a panoramic scene, wherein the object The object information of the identifiable object detected by the information acquisition system is displayed in a panoramic manner in an augmented reality manner. It should be specially pointed out that the reflective surround display device herein is preferably based on the Chinese utility model patent ZL201720316246.9 and the Hong Kong short-term patent HK1229152, the inventor is Luo Zhenbang, and the patentee is the solution to the blade (Solved) By Technology Co., Limited) is a reflective surround display system.

11 is a schematic diagram of a robot with a fisheye camera incorporating a 360 degree surround orientation and position sensing object information acquisition system according to an embodiment of the present invention, respectively showing a fisheye camera and three fisheyes. Camera robot. a robot located on the left side of FIG. 11 , which is provided with a movable fisheye image capturing unit 10, which can change the position of the fisheye camera on the robot to take a plurality of fisheye images, which can be The moving fisheye image capturing unit 10 is capable of photographing a plurality of fisheye images in the environment from different positions in the vertical direction as a positioning operation; referring to FIG. 11, the robot located on the right side in FIG. 11 has three positions on the robot a fixed fisheye image capturing unit 10, the three fisheye image capturing units 10 are sequentially arranged in a vertical direction, wherein the upper two fisheye image capturing units 10 have higher shooting for a higher altitude environment The resolution, the lower fisheye image capture 10 has a higher resolution for lower altitude environments and has additional, more accurate measurements as a positioning operation.

Referring to FIG. 10, the remote shopping system with 360 degree surround orientation and position sensing object information acquiring system based on 360 degree digital fisheye image, 360 degree surrounding orientation and position sensing object information of the above embodiment. The acquisition system can be set in a store remote shopping front-end device (above, referring to FIG. 10, the store remote shopping front-end device may specifically be a robot, and the robot can be wirelessly connected with the panoramic augmented reality device, for example, the robot can communicate with the mobile communication network The panoramic augmented reality device is connected, and the fisheye image capturing unit 10 is disposed on the robot. The fisheye image capturing unit 10 can be arranged with reference to the setting shown in FIG. 11, and the products in the store are respectively labeled with a double square pattern QR code label. Each store has a unique two-square pattern QR code label with a pan/tilt optical telescope including a fisheye camera that is used to stretch and capture a two-dimensional code located further away.

The plurality of fisheye images captured by the fisheye image capturing unit 10 of the above robot are processed to form a fisheye panoramic augmented reality image, which is transmitted to a remote client computer and passed through a reflective surround display device. Display, so that users can feel the product information in the remote store immersively without leaving the house, which enhances the user experience.

Referring to FIG. 10, the remote shopping system of the embodiment of the present invention has the following work flow: a remote network (such as the Internet) customer communicates with a customer service through a network (such as the Internet), and selects a robot with a digital fisheye camera, and the robot automatically Walking between the merchandise display racks, the fisheye camera takes a realistic fisheye panorama of the merchandise, and the remote network client is equipped with a computer with a reflective surround display device to output a realistic fisheye panoramic image, which can be accessed by a remote network (eg internet) customer After selecting a specific item in a remotely-spaced remote store front-end device, a remote network (eg, Internet) customer selects a specific item, and the fisheye camera identifies a double square label pattern with a QR code on the specific item and accesses The network (such as the Internet) obtains information feedback from the network (such as the Internet) for customer reference, and calculates what direction and orientation the product is in the robot. If a remote network (such as the Internet) customer selects the item, the robot accurately places the item into the shopping cart based on the calculated orientation and direction, and so on, and selects other items. Finally, the remote network (such as the Internet) customer pays the relevant payment through the remote payment function, and the mall mails the goods or personally to the remote network (such as the Internet) customer residence. By using the remote shopping system of the present invention, the reality is enhanced, the customer can realize the immersive shopping without leaving the door, and the article manufacturer information acquisition unit 30 can accurately obtain the manufacturer and the expiration date of the product from the network (for example, the Internet). , production batches and other information, to achieve intelligent shopping.

The robot with 360-degree surround orientation and position-aware object information acquisition system based on 360-degree digital fisheye image can also be used in the express sorting system to accurately classify the express.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the invention. In the present specification, the schematic representation of the above terms does not necessarily mean the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Although the embodiments of the present invention have been shown and described, it is understood that the foregoing embodiments are illustrative and not restrictive Variations, modifications, alterations and variations of the above-described embodiments are possible within the scope of the invention.

Claims

A 360-degree circumferential orientation and position sensing object information acquiring system based on a 360-degree digital fisheye image, characterized in that: a 360-degree fisheye image obtained by each fisheye observation point of the fisheye image capturing unit, The 360-degree surrounding object detection is performed by an image processing method on the overall environment surrounding the fisheye observation point of the system to sense the 360-degree orientation of the object and the 360-degree spatial distance and position of the perceived object, the 360-degree circumferential orientation. And the location-aware object information acquisition system includes:

a fisheye image capturing unit that captures a surrounding environment to be recognized using at least one fisheye digital camera, the optical axis of the fisheye digital camera being perpendicular to a viewing plane to obtain the to-be-identified An overall fisheye image of the environment's 360 degree surround;

The object detecting unit receives the output image of the fisheye image capturing unit, and detects at least one identifiable object from the overall fisheye image according to different object categories;

An object information acquiring unit, wherein the object information acquiring unit is configured to scan the image of the label object in the overall image to extract the included object information, but not if the object type is a label type identifiable object; It is limited to the network information address of the object, and accesses the object network information address in real time to obtain the network information of the object. If the object category is a non-label type identifiable object, the object image is used to identify the object in real time through the network image recognition service. Obtaining identification object information;

An object orientation computing unit, the object orientation computing unit receiving the detection result of the object detection unit, and determining, for each identifiable object detected by the object detection unit, each of the identifiable objects Corresponding position points in the overall fisheye image, and calculating a 360 degree orientation of each of the identifiable objects relative to the fisheye camera in the environment to be recognized according to imaging parameters of the fisheye camera ;as well as

a system positioning operation unit, the system positioning operation unit receiving an operation result of the object orientation operation unit, configured to be based on at least two different sets of identifiable objects provided by the object orientation operation unit with respect to the fisheye camera Direction information, based on triangulation, calculating a 360-degree spatial distance and position of the identifiable object relative to the system, and obtaining a global or local environment of the identifiable object from the object information acquiring unit or the information contained in the object The absolute position of the map, through the surrounding observation environment, the objects that provide positioning information, calculate the absolute position of the system's global or environmental map.
The 360 degree surround orientation and position sensing object information acquiring system according to claim 1, wherein:

The tag type identifiable object is a specially customized easy-to-recognize tag pattern; the object information acquiring unit obtains a network channel bound to the identifiable object by partially scanning an object code on the identifiable tag pattern, and The object information is obtained by accessing a network channel.
The 360 degree surround orientation and position sensing object information acquiring system according to claim 1, wherein:

The non-label type identifiable object is a specific object contour pattern preset by the system, and the object detecting unit performs object detection of the entire image by using an artificial intelligence image service built in or through the network.
The 360 degree surround orientation and position sensing object information acquiring system according to claim 2, wherein:

The identifiable label pattern is a high contrast double square pattern, and the object code is a two-dimensional code for obtaining an object network channel pasted or printed in the double square pattern.
The 360-degree surround orientation and position sensing object information acquiring system according to claim 4, wherein:

A Morse code that encodes the basic information of the identifiable object is overlaid on the double square pattern.
The system of claim 1 , wherein the method further comprises:

An optical telescope unit connected to the system positioning operation unit for digital shooting for automatically aiming and/or focusing the identifiable object according to the relative positioning information provided by the system positioning operation unit And digital shooting.
The 360 degree surround orientation and position sensing object information acquiring system according to claim 1 or claim 6, further comprising:

An object information output unit, the object information output unit is connected to the system positioning operation unit and/or the object information acquisition unit, and the object information output unit comprises a display unit and/or a voice unit;

The display unit is configured to visually display at least one of direction information, positioning information, and object information of the identifiable object; the voice unit is configured to audibly broadcast direction information and positioning of the identifiable object At least one of information and object information.
The 360 degree surround orientation and position sensing object information acquiring system according to claim 1, wherein:

In the fisheye image capturing unit, the number of the fisheye cameras is one, which is configured to perform photographing at at least two different positions, so that the object orientation arithmetic unit can provide at least two different sets of relative Direction information of the fisheye camera.
The 360 degree surround orientation and position sensing object information acquiring system according to claim 1, wherein:

In the fisheye image capturing unit, the number of the fisheye cameras is two or more positions that are relatively fixed in position, so that the object orientation computing unit can provide at least two different sets of cameras relative to the fisheye. Direction information.
The 360 degree surround orientation and position sensing object information acquiring system according to claim 1, wherein the object information acquiring system further comprises a remote output unit, and the remote output unit comprises a remote panoramic augmented reality device. The panoramic augmented reality device includes a reflective surround display device, and an overall fisheye image of the environment to be recognized received from the object information acquisition system is displayed on the reflective surround display device in a panoramic scene, wherein The object information of the identifiable object detected by the object information acquiring system is displayed in the panoramic scene in an augmented reality manner.