JP2007098567A - Autonomous control type robot and its control device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an autonomous control type robot capable of making an emergency responding control. <P>SOLUTION: The autonomous control type robot is structured as capable of making the emergency responding control by monitoring the safety of the robot operation through an independent system provided separately from the autonomous control system. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an autonomous control robot and a control device therefor.

  As a method for generating 3D-CAD data based on image information generated by photographing with an imaging device, there is a method for obtaining three-dimensional coordinates from a plurality of photographs. For example, there is a software called 3D Builder, which was developed at 3D Construction Company in the United States and sold in Japan by K.G. When a person inputs corresponding points between multiple photos one by one, the point and the position and orientation of the photographed camera are calculated as 3D coordinates, and CAD data can be created interactively or texture mapping can be performed. Can be done.

Also, in Japanese Patent Laid-Open Nos. 2002-32741 and 2002-32742, if feature points (windows) are designated in an initial image or feature points are first set based on feature point position setting data, they are continuously recorded. Automatically track feature points in the input image information so that humans do not have to input corresponding points one by one, calculate three-dimensional coordinates, and connect the point data to perform texture mapping The concept of a system that can do is shown.
JP 2002-32741 A JP 2002-32742 A Software sold by 3D Builder, KK TG

  However, it is not necessary for a person to input and set corresponding feature points in a plurality of photographs, or accurate CAD data is generated without texture mapping, or the imaging camera is actively moved to perform CAD. We will improve the accuracy by generating data more accurately and by generating it with reasonable accuracy so that it can be used for robot control etc. and moving the imaging camera back and forth, left and right It has not been considered to be able to do this in real time or to integrate CAD data generated by moving a plurality of imaging cameras. In addition, it can be applied to visual guidance control devices for automatic machines such as robots and navigation systems such as automobiles, in combination with a function for searching for arbitrary CAD data after or while generating CAD data for as-built. It was not considered to be applied to control robots.

  An object of the present invention is to provide an autonomous control type robot capable of emergency response control and its control device.

The present invention is basically characterized in that, in an autonomous control type robot, the operation safety of the robot can be monitored by an independent system different from the autonomous control system,
In addition, in the control device of the autonomous control type robot, the operation safety of the robot can be monitored by an independent system different from the autonomous control system,
In addition, in an autonomous control robot, when emergency response is required during safety monitoring of the robot, the emergency response control of the robot can be performed by an independent system different from the autonomous control system. age,
In addition, in an autonomous control robot control device, emergency response control of the robot can be performed by an independent system separate from the autonomous control system when emergency response is required during robot safety monitoring. It is characterized by that.

  As described above, according to the present invention, when an emergency response is required during safety monitoring of the robot in the autonomous control type robot or its control device, the emergency control of the robot is performed independently of the autonomous control system. Since response control is possible, in the event of an emergency, an emergency stop or switching to manual can be performed reliably.

  The present invention realizes a visual information processing apparatus and an application system that can achieve the object by combining the following means alone or selectively.

The first means includes an image information processing device that inputs image information obtained by photographing an object with a movable imaging device and generates CAD data of the object, and a storage device that stores the generated CAD data. In the visual information processing apparatus, the image information processing apparatus includes an update target determination processing function for determining whether or not the generated new CAD data is an update target of already generated CAD data, and an existing target to be updated. A CAD data accuracy comparison processing function for comparing the accuracy of CAD data and new CAD data, and CAD for updating existing CAD data with new CAD data when the accuracy of new CAD data is higher than the accuracy of existing CAD data to be updated It is characterized by having a data update processing function.
This first means has a function in the visual information processing device that receives image information obtained by a movable imaging device as an input and processes the input image to generate CAD data of an object existing in the moving space. CAD data of various objects existing in a moving space can be easily generated.

  Further, when the generated new CAD data is an update target determination function for determining whether or not the existing CAD data is already generated, and when the new CAD data is an update target of the existing CAD data, When the accuracy of the CAD data is compared with the accuracy of the existing CAD data generated before the update target and the accuracy is improved, a function for updating the CAD data is also provided. By approaching and obtaining image information of a more detailed portion of the object, it becomes possible to update while generating more detailed CAD data of the object. In addition, since the moving imaging apparatus can input image information of the top, bottom, left, and right sides and the back side of the object, not only the color and shape of the front side of the object, but also the top and bottom, left and right sides, and even the color and shape of the back side can be CAD. It can be generated as data. The update target determination processing function for determining whether or not the new CAD data is an update target of the existing CAD data is, for example, an update target when there is no previously generated CAD data at the same location after the CAD data is generated. CAD data is added as it is, and if it exists, it is to be updated. When the new CAD data is an update target of the existing CAD data, the accuracy of the new CAD data is compared with the accuracy of the existing CAD data to be updated, and when the accuracy is improved, by updating the CAD data, Even if CAD data of the same part is generated each time, it is not updated except when the accuracy is improved, and is updated when the accuracy is improved. By repeating this processing, The accuracy of the object existing in the moving space is improved every time the CAD data is updated, and finally the CAD data with high accuracy can be obtained. As described above, if only the update that improves the accuracy of the CAD data is performed, the CAD data is generated with a certain degree of accuracy from the image information obtained when the imaging apparatus is away from the target. However, according to the image information taken by the imaging device close to the object, the generation accuracy can be improved accordingly.

  Even in operation, if the accuracy of the CAD data is not good, for example, the CAD data can be used for rough positioning control with respect to a distant object. Since it is possible to perform positioning control with higher accuracy by using the CAD data, CAD data can be continuously and effectively used even from the first stage with lower accuracy.

  In addition, with such a CAD data update function, a lot of similar CAD data is not accumulated regardless of how many times CAD data is generated for the same part of a non-moving object. If a large number of objects are generated, it is a case where moving objects are included. Therefore, the movement of the moving target object such as smoke or steam moves by tracking and so on. Tracking of spotlights or moving shadows can be easily performed by referring to the update history because CAD data is sequentially updated.

  The second means is a function of automatically generating or automatically updating the CAD data of the first means, and the three-dimensional coordinate data is obtained by a photometric method in which corresponding points are related by tracking and tracking feature points of image information. Is to ask for.

  Such a second means uses a photo measurement method in which corresponding points are related by tracking the feature points of the image information to the function of automatically generating or automatically updating the CAD data of the first means. Since the three-dimensional coordinate data is obtained, the image information obtained by photographing with the moving imaging device is obtained as a plurality of pieces of image information obtained by photographing the object from various viewpoints, and is input during the movement. Since the information becomes continuous and the feature points of the continuous image information can be continuously tracked and tracked, it is possible to perform photo measurement in which the corresponding points are correctly associated without losing sight of the feature points. The automatic update of CAD data is performed in the same place after the automatic generation, and if there is data before it is determined that the data generated this time is the data generated last time, it is repeatedly performed. As a result, the CAD data can be automatically updated. If the CAD data can be automatically updated, the CAD data is generated with a certain degree of accuracy from the image information obtained when the imaging device is away from the target object, and the imaging device approaches the target object. According to the image information photographed in this way, the CAD data generation accuracy can be improved to some extent. If the accuracy of the CAD data is not good in this way, for example, rough positioning control for a distant target object is performed. Since it is possible to perform positioning control with high accuracy based on CAD data with higher accuracy if it is nearer, the CAD data is effective even from the first stage of CAD data generation. It becomes possible to use it. In addition, if there is an update function of CAD data, the CAD data is sequentially updated for the movement of the moving object such as smoke and steam, the moving spotlight or the moving shadow, etc., so refer to the update contents. By doing so, it becomes possible to carry out easily.

  The third means is a function of automatically generating or automatically updating the CAD data of the first and second means in a three-dimensional stereo measurement method in which corresponding points are related by tracking and tracking feature points of an image. The coordinate data is obtained.

  Such a third means is a function for automatically generating CAD data of the first and second means, and obtains three-dimensional coordinate data by a stereo measurement method. If the three-dimensional coordinate data of the object is obtained from the two pieces of image information obtained by the apparatus by the stereo measurement method, the object being photographed while being positioned in front of the image pickup apparatus before the two image pickup apparatuses are moved. Since the three-dimensional coordinate data can be obtained, the CAD data of the object being photographed while being positioned in front can also be obtained from the image information before moving the imaging device. If it is possible to obtain CAD data before moving the imaging device, if the target object is visible from the beginning when applied as a visual guidance control device for an automatic machine such as a robot, the imaging device is moved. Thus, it is possible to omit the time for generating CAD data, and to make it possible for an automatic machine such as a robot to access the target object. If the CAD data can be automatically updated, the CAD data is generated with a certain degree of accuracy when the imaging device is far from the target object, and the generation accuracy is moderate when the imaging device is close to the target object. Can be raised. In this way, if the accuracy of the CAD data is not good, the CAD data can be used, for example, for rough positioning control with respect to a distant target object. Since it can be used to perform more accurate positioning control based on the raised CAD data, the CAD data can be used effectively even from the first stage. In addition, if there is an update function of CAD data, the CAD data is sequentially updated in accordance with the movement of the moving target object such as smoke or steam, or the moving spotlight or the moving shadow. By referring to, it becomes possible to carry out easily.

  The fourth means has a function of automatically generating CAD data of an object existing in a moving space by processing image information obtained by a movable imaging device, and a function of automatically generating this CAD data. In this method, three-dimensional coordinate data is obtained by a stereo measurement method in which corresponding points are related by tracking and tracking feature points of image information.

  Such a fourth means has a function of automatically generating CAD data of an object existing in a moving space by processing image information obtained by a movable imaging device in a visual information processing device. As in the first and second means, it is possible to easily obtain CAD data of various objects existing in the moving space, and to obtain three-dimensional coordinate data by the stereo measurement method. Therefore, for example, if the three-dimensional coordinate data of an object is obtained by the stereo measurement method from two pieces of image information obtained by two imaging devices, the two imaging devices are moved before the two imaging devices are moved. Since the three-dimensional coordinate data of the object being photographed at the front can be obtained based on the image information before moving the imaging device, the CAD data of the object being photographed by being located at the front It is possible to obtain the image information before the image pickup device moves. If it is possible to obtain CAD data before moving the imaging device, if the target object is visible from the beginning when applied as a visual guidance control device for an automatic machine such as a robot, the imaging device is moved. Thus, the time for generating CAD data can be omitted, and an automatic machine such as a robot can access the target object. In this system, even if there is no automatic update function of CAD data, if the photographing distance is set to a constant interval and CAD data of an object is generated with a predetermined accuracy, only automatic generation is performed with a simpler system. Can be easily realized and used.

  A fifth means is to make the CAD data distinguish a contour line from an actual line in a visual information processing apparatus configured by selectively applying the first to fourth means.

  In such a fifth means, in the visual information processing apparatus configured by selectively applying the first to fourth means, the CAD data distinguishes the contour line from the actual line. Therefore, when capturing the shape data of the object, since the line detected as the contour line is also a line indicating the shape of the object, it can be stored as shape data. Further, when a line actually exists like a pattern line, the line is used as shape data having information on an absolute position, or is information on a pattern forming the pattern of the object. Therefore, if the contour line and the actual line are distinguished and managed in the generated CAD data, if you want to handle the shape data, use more information using both the contour line and the actual line. An accurate shape can be recognized and processed.

  On the other hand, when trying to search for patterns that are pattern matching, etc., if the contour line data is handled together, lines that are not in the pattern are included, so it is difficult to search without correct matching, but in that case If a pattern to be searched is matched using only actual line data, the pattern can be easily searched. Since the contour line and the actual line are distinguished and managed, only the necessary line can be efficiently used when necessary. Accordingly, CAD data can be efficiently generated and searched even in an environment where a patterned object and an unpatterned object are mixed. Also, the spotlight or shadow outline is an outline if it is known as the outline of the surface that is illuminated or the outline of the shadow that is not illuminated, but there is an object that blocks the light from the light source or the light source. Since it has both features that can be handled as a pattern because it does not change if the image pickup device moves only if it does not move, the distinction content may be changed from the actual (pattern) line to the contour line when it is known, If it is known that the obstructing object moves but does not move at the time interval using CAD data, the shadow and other contour lines are considered as other contour lines in consideration of the case where the shadow is used as a pattern matching pattern. Alternatively, management may be performed separately from actual lines.

  A sixth means is a visual information processing apparatus configured by selectively applying the first to fifth means, wherein the CAD data is constituted by points, lines, and plane data.

  In such a sixth means, in the visual information processing apparatus configured by selectively applying the first to fifth means, the CAD data is constituted by points, lines, and plane data. For this reason, it is possible to eliminate the need to replace the data with solid data such as a cylinder or a cube, and the processing can be simplified and speeded up easily. Also, when updating the CAD data, a complicated process for updating the solid data is not required, so that the update process can be easily realized. Furthermore, since CAD data is generated based on image information obtained while moving the imaging apparatus, when the target object is photographed from a distance, the dice and the like look like a cube, so the CAD as a cube Although data is generated, when the imaging device is close, the corners of the dice are rounded, and strictly speaking, they are not complete cubes. This is true for all surrounding environments. Accordingly, the minimum necessary unit of CAD data when generating CAD data based on image information obtained by imaging with the moving imaging apparatus is sufficient for point data, line data, and plane data. is there. Even if you generate solid data, if you look at it from a closer distance, it will not be the solid you originally defined. Even if the building seen from a distance is like a rectangular parallelepiped, if the imaging device moves and enters the building, the building that initially appeared as a rectangular parallelepiped is composed of flat walls, If you look closely at the wall, you can see that it is composed of an assembly of a plurality of planes. Even if it is first replaced with solid data such as a rectangular parallelepiped, CAD is based on image information captured by a moving imaging device. When data is generated, it is necessary to update immediately. Accordingly, the processing can be easily performed by generating and updating the minimum necessary CAD data. In the case of this system, it is the most suitable configuration of CAD data.

  The seventh means is to allow at least the CAD data of the line to be expressed as a function in the visual information processing apparatus configured by selectively applying the first to sixth means.

  In such a seventh means, in the visual information processing apparatus configured by selectively applying the first to sixth means, at least the CAD data of the line is expressed as a function. If CAD data is expressed by an equation of a straight line connecting a start point and an end point, no matter how long a straight line is, it is not necessary to increase the point data only by changing the coordinate values of the start point and the end point. Also, by defining some representative points that pass through the start point, end point, and halfway in the curve, and expressing them with functions that fit to those points, for example, a curve is displayed as a collection of short line data Compared to the case, the number of point data and line data is reduced, and the number of elements (line data) constituting the surface data is also reduced. Therefore, even when the CAD data is increased, the memory capacity of the CAD data is saved. It becomes possible.

  The eighth means is to provide color information at least on the CAD data of the surface in the visual information processing apparatus configured by selectively applying the first to seventh means.

  In such a visual information processing apparatus configured by selectively applying the first to seventh means, color information is provided at least on the CAD data of the surface. Matching search using can be performed. If all object boundaries are considered as boundary lines, the idea that color information is only in surface data, not line data, can be applied to the system. That is, when CAD data is generated based on image information obtained by imaging with a moving imaging device, a portion that is visible as a line becomes surface data having a long and narrow area when viewed in the vicinity. Of course, the line data can be provided with color information when it looks like a line. If at least surface data has color information, CAD data that can be applied to matching using a color pattern or red, blue, and yellow recognition of a traffic signal can be generated.

  A ninth means is to provide brightness information at least in the color information of the CAD data of the surface in the visual information processing apparatus configured by selectively applying the first to eighth means.

  In such a ninth means, in the visual information processing apparatus configured by selectively applying the first to eighth means, brightness information is provided at least in the color information of the CAD data of the surface. So, for example, when trying to recognize the traffic signal status using the CAD data generated by this system, the lamp type traffic signal will show a red color even when the red signal lamp is not lit. Therefore, it is difficult to recognize whether or not the signal lamp is lit only by color information, but if there is brightness information, whether or not the lamp is lit is turned on and off. You can easily recognize the brightness when you are comparing. Of course, in the case of monochrome processing, the color information is grayscale, but it is also information on brightness. In addition, if a surface bright enough to affect other target objects is known in advance as a light source, if there is a shadow caused by the influence of the light source, it is geometrically determined that the shadow is caused by the light source. Therefore, it is possible to distinguish a surface with a certain brightness or more as a light source and CAD data. At this time, for easy use later, if there is a difference in brightness for each wavelength component or brightness depending on the direction of radiation, information related to the distribution characteristics is also generated in the CAD data. It is good to keep.

  A tenth means is a visual information processing apparatus configured by selectively applying the first to ninth means so that the CAD data has information relating to the time at which the CAD data was generated.

  In such a tenth means, in the visual information processing apparatus configured by selectively applying the first to ninth means, the CAD data is provided with information relating to the time generated. When the generated CAD data is referred to, the time when the CAD data is generated can be referred to. Therefore, for example, when there is a difference when comparing the CAD data generated at the same place with the CAD data generated before, the current time and the time registered in the previous CAD data are It is possible to recognize how many hours ago the state has changed. When referring to CAD data generated while generating CAD data, if the generation time is in the CAD data, it can be confirmed which is the CAD data generated earlier.

  The eleventh means is to leave the previous CAD data when the CAD data is updated in the visual information processing apparatus configured by selectively applying the first to tenth means.

  In such a eleventh means, in the visual information processing apparatus configured by selectively applying the first to tenth means, the previous CAD data is left when the CAD data is updated. It becomes possible to leave the process of changing the CAD data. If the remaining CAD data includes time data, the movement of the CAD data is calculated based on the movement amount of the time-series CAD data and the interval at which the CAD data calculated from the reference time is generated. It is also possible to estimate and recognize the speed.

  A twelfth means is to delete old CAD data at a predetermined timing when updating the CAD data in the visual information processing apparatus configured by applying the eleventh means.

  In such a twelfth means, in the visual information processing apparatus configured by applying the eleventh means, the old CAD data is deleted at a predetermined timing when the CAD data is updated. In other words, the previous CAD data may be left at the time of update by limiting the amount of data left by, for example, leaving data up to 5 times before or leaving data up to 10 seconds ago. The amount of CAD data does not diverge and the memory capacity can be saved.

  According to a thirteenth means, in the visual information processing apparatus configured by selectively applying the first to twelfth means, the CAD data is grouped and distinguished from data determined to move together. Is to do.

  In such a thirteenth means, in the visual information processing apparatus configured by selectively applying the first to twelfth means, the CAD data is grouped with data determined to move together. Since the CAD data generated from the moving objects are moving together, grouping them as a group of the data moving together, the CAD data constituting the moving object is converted into a group. It becomes easy to manage by grouping. By grouping the data such as the light reflected on the surface, the movement of the virtual image reflected in the mirror surface, the illumination of the spotlight, the shadow, etc., tracking and updating of the CAD data can be performed by the CAD data of the group. Therefore, processing can be facilitated.

  In a visual information processing apparatus configured by selectively applying the first to thirteenth means, the fourteenth means moves the CAD data attached to the CAD data of the surface separately from the CAD data of the surface. By detecting this, it is made to distinguish as CAD data of a virtual image.

  In such a fourteenth means, in the visual information processing apparatus configured by selectively applying the first to thirteenth means, the CAD data attached to the CAD data of the surface is the CAD data of the surface. Since it is distinguished as CAD data of the virtual image by detecting the movement separately, it is managed by distinguishing the CAD data actually on the surface from the virtual image that happens to appear on the surface by reflection. can do. If the target object is recognized, the process of recognizing the target object by performing pattern matching with the pattern data of only the original target object, excluding the virtual image data that happened to be captured, is recognized. It becomes easy. What is referred to as a virtual image herein is referred to as a virtual image in the sense that image information that is seen through to the side facing the virtual image reflected on the mirror surface is not directly attached to the target surface data. Also, spotlight illumination or shadows may be defined as moving virtual images.

  In a visual information processing apparatus configured by selectively applying the first to fourteenth means, a fifteenth means is a mirror surface or a surface through which the CAD data of the surface with the virtual image CAD data is attached. It is to be distinguished as.

  In such a fifteenth means, in the visual information processing apparatus configured by selectively applying the first to fourteenth means, the CAD data of the surface with the virtual image CAD data is converted into a mirror surface or Since it is distinguished as a transparent surface, it can be understood that a virtual image appears on the surface after the surface is once distinguished. For example, CAD data of a new point, line, or surface is attached to the surface. When such data is generated, if the surface is a mirror surface, an object corresponding to a real image that is a virtual image is newly generated from the imaging device, the viewpoint, and the position and orientation of the surface. By examining the CAD data in the direction of the space, the reflected virtual image can be easily confirmed to be a new virtual image generated by specular reflection. Similarly, in the case of a transmissive surface, it is possible to easily confirm that it is a virtual image of a previous entity that can be seen through by examining the direction of the previous space that can be seen through. Therefore, for the time being, even if it is not distinguished whether it is a mirror surface or a transmission surface, if it is possible to distinguish between a surface where a virtual image is reflected and a surface where a virtual image is not visible, whether the processing considering the virtual image is performed when the surface is updated Since it can be discriminated whether or not, processing becomes easy.

  A sixteenth means is a visual information processing apparatus configured by selectively applying the first to fifteenth means, and reflects when a transparent surface or a surface determined to be a mirror surface is assumed to be a mirror surface. When CAD data that is determined to be a real image of the virtual image exists at a position corresponding to the previous virtual image, the surface is identified as a specular surface.

  Such a sixteenth means is a visual information processing apparatus configured by selectively applying the first to fifteenth means, assuming that the surface with the virtual image CAD data is a specular surface. Since there is a real image (entity) of the virtual image at a position corresponding to the virtual image, when such CAD data exists, the surface is distinguished as a mirror surface. If there is no entity, it is understood that the surface is a transparent surface, and can be managed as a transparent surface. If the entity exists, it is confirmed that the surface is a mirror surface, and the surface is a mirror surface or a half mirror surface. Therefore, by performing such processing, it becomes possible to easily identify whether or not the surface data is a mirror surface (mirror surface or half mirror surface). If there is no distinction between the surfaces in this way, each time the surface is newly added, for example, paint such as paint, or a virtual image that is reflected in the mirror surface, it can be seen through. Since it is not possible to perform final recognition without confirming whether or not the virtual image is present, the processing takes extra time. Therefore, the fact that the surface has been identified eliminates at least the confirmation of the mirror surface and the confirmation of the transmission surface, so that the virtual image can be recognized efficiently. .

  In a visual information processing apparatus configured by selectively applying the first to sixteenth means, the seventeenth means causes at least one of the surface CAD data to have reflectance, transmittance, or both. That is.

  In such a seventeenth means, in the visual information processing apparatus configured by selectively applying the first to sixteenth means, the CAD data of the surface has either or both of reflectance and transmittance. Therefore, for example, when the reflectance is 100% and the transmittance is 0%, the surface is assumed to be a perfect mirror surface, and when the reflectance is 50% and the transmittance is 50%. Assuming that the surface is a half mirror surface, the following processing can be performed. When this CAD data is generated, the actual reflectance or transmittance may be obtained from the luminance (brightness) information acquired from the imaging device, and the numerical value may be given, whether the reflectance is 100% or not. Only simple identification of mirror surface or half mirror may be performed by identifying whether the rate is 50% or not. Incidentally, in the case where information on reflectance and transmittance is provided, a system that handles the difference between the sum of both values and 100% as the absorption rate can be easily realized by using such CAD data. Will be able to. Furthermore, strictly speaking, the definition of reflectance or transmittance may differ for each wavelength component, so it may be defined for each wavelength regardless of how finely it is defined as CAD data. Moreover, you may make it define an emissivity for every wavelength. In addition, the emissivity and reflectivity may have directionality in the direction, so the state and shadow of the surface caused by the light source, and the state and shadow of the surface caused by the light source reflected on the object in the environment. The CAD data should also have directivity characteristics so that it can be easily recognized from the geometric relationship. The directivity of the reflectance may include an element indicating the roughness of the reflecting surface. For example, using the characteristic that the sharpness of a virtual image varies depending on the roughness of the reflecting surface, the sharpness (blurring) of the contour of the reflected virtual image is extracted by processing raw image information, which is then extracted from the rough surface. This should be taken as an index equivalent to that.

  According to an eighteenth means, in the visual information processing apparatus configured by selectively applying the first to seventeenth means, the surface where the color of the CAD data changes when the viewpoint position of the imaging device changes is such a surface. It is to distinguish between being.

  In such a visual information processing apparatus configured by selectively applying the first to seventeenth means, the eighteenth means is such that the color of the CAD data changes when the viewpoint position of the imaging device changes. Since such a surface is distinguished, the surface whose color changes depending on the viewing angle can be identified. If it is identified that such a surface is such a surface, for example, when trying to specify using the CAD data of an orange car among many cars, the color of the CAD data on the surface of the car is changed. A car that happens to appear orange when it is generated can be prevented from being erroneously identified as simply an orange car. In other words, if it is discriminated that the surface changes color depending on the viewing angle, the color information of the CAD data generated when viewed from other angles is also confirmed, and it is really an orange car. It becomes possible to specify.

  According to a nineteenth aspect, in the visual information processing apparatus configured by selectively applying the first to eighteenth means, a portion that does not change in the input image information even if the imaging apparatus moves is a target of processing. Is to be outside.

  In such a nineteenth means, in the visual information processing apparatus configured by selectively applying the first to eighteenth means, the portion of the input image information that does not change even when the imaging apparatus moves is Since it is excluded from the processing target, for example, dust and dust adhering to the lens of the imaging device, traces of radiation deterioration of the CCD element, characters superimposed on the image information, etc. In fact, an object that does not exist in the environment space is tried to be converted to CAD data by trying to convert it into CAD data, or a contradiction occurs when trying to convert it to CAD data. It is possible to efficiently perform processing only on necessary information without performing useless processing such as knowing that the noise component does not exist in the space.

  In a visual information processing apparatus configured by selectively applying the first to nineteenth means, the twentieth means is the shape, size, density, directionality, and color for the image information obtained by the imaging apparatus. The relevant part in the image information corresponding to the pattern is extracted with a geometric pattern such as, a temporal pattern such as a generated cycle, or both the geometric pattern and the temporal pattern. The extracted portion is excluded from processing.

In such a twentieth means, in the visual information processing apparatus configured by selectively applying the first to nineteenth means, the shape, size, density, In a geometric pattern such as directionality, color, etc., in a temporal pattern such as a generated cycle, or in both the geometric pattern and temporal pattern in the image information corresponding to the pattern Since the part is extracted and the extracted part is excluded from processing, electrical noise included in the image information obtained by the imaging device or radiation noise that enters randomly can be easily processed. Can be excluded. In other words, the noise component included in the image information stage may be removed, but the image information generated in the part at the address of the part, for example, the address of the image memory, etc., before it is removed is noise. Can be removed in advance and excluded from the processing target. Electrical noise or random incoming radiation noise does not change in frequency and magnitude at the time of occurrence, but changes in geometry such as shape, size, density, directionality, and color. Since there is a temporal feature such as a target feature and a generated cycle, it can be recognized relatively easily from the feature at the preprocessing stage.

  In a visual information processing apparatus configured by selectively applying the first to twentieth means, the twenty-first means is a geometric pattern such as shape, size, and color for the generated CAD data. Or the generated CAD data corresponding to the geometric pattern and the temporal pattern is extracted with a temporal pattern such as a generated cycle or both, and the extracted CAD data is processed. It is to be excluded from the target.

  Such a twenty-first means is a visual information processing apparatus configured by selectively applying the first to twentieth means, such as geometry such as shape, size, color, etc. for generated CAD data. The CAD data generated by extracting the CAD data corresponding to the pattern in a periodic pattern, in a temporal pattern such as a generated cycle, or in both the geometric pattern and the temporal pattern, and extracting the CAD data Is excluded from the processing target, so dust that is flying in the space, outdoor rain, snow, etc. can be removed as a noise component if necessary, and as a result Thus, it is possible to efficiently handle only CAD data that actually exists in the space. If a specific example is described, when CAD data is generated based on image information obtained by photographing the outdoors with a stereo imaging device, rain or snow CAD data is also generated. When only the CAD data of a running car is used for car recognition, if CAD data such as rain around the car exists, it becomes a noise component in the car recognition process, and the efficiency is deteriorated. However, by including this processing, before the vehicle recognition processing, specific noise, for example, linear CAD data arranged in a short vertical direction is first distinguished or deleted because it is rain CAD data. Thus, it is possible to easily recognize the vehicle by eliminating the rain CAD data, which becomes noise in the vehicle recognition process.

  The twenty-second means is a visual information processing apparatus or a visual information processing system to which the first to twenty-first means are applied, and a plurality of image information processing for inputting and processing image information obtained by a plurality of imaging devices. It is to provide a device and integrate CAD data generated and updated by each.

  In the visual information processing apparatus configured by selectively applying the twenty-second means, the first to twenty-first means, or the visual information processing system to which the visual information processing apparatus is applied, image information obtained by a plurality of imaging apparatuses is used. Since the CAD data generated and updated by each of the plurality of image information processing apparatuses is integrated based on the CAD data generated based on the image information obtained by photographing various places with the plurality of imaging apparatuses. It can be effectively shared with each other. Moreover, although it takes time to generate a CAD data by photographing a wide area with one imaging device, it is possible to integrate the respective CAD data by photographing with a plurality of imaging devices to generate CAD data. A wide range of CAD data can be generated in a short time.

  A twenty-third means integrates CAD data generated and updated by a plurality of image information processing apparatuses in a visual information processing apparatus configured by applying the twenty-second means or a visual information processing system to which the twenty-second means is applied. The means is to allow mutual image information processing devices to input and output mutual CAD data.

  Such a 23rd means is a CAD data generated and updated by each of a plurality of image information processing apparatuses in a visual information processing apparatus configured by applying the 22nd means or a visual information processing system to which the visual information processing system is applied. Since the visual information processing devices can input and output the mutual CAD data, the visual information processing devices can directly exchange and process data. Specifically, for example, if mutual visual information processing devices are connected by a communication line, a command for requesting CAD data of a predetermined area via the communication line is prepared, and based on the command, The CAD data generated by each other can be obtained, or the CAD data generated by the other party can be transmitted to the other party. When sending / receiving CAD data, it is easier to know the amount of relative difference in the world coordinate system that is the reference when generating the CAD data, or to use the same coordinate system. If the correspondence of the world coordinate system is not taken in advance, it is necessary to find a matching part between the acquired CAD data and the CAD data generated by the user and obtain a shift between the world coordinate systems. If CAD data is not generated, a matching location cannot be obtained. As a method for matching the coordinate systems of each other, for example, before the image pickup apparatus is moved for the first time, the coordinates of the place and the attitude of the image pickup apparatus are set with numerical values in the same world coordinate system, so that they can be easily matched. Is possible. However, there is an error that is set first in Hokkaido and Kyushu in Japan, and the deviation when moving together and coming together in Tokyo will be caused by that error, so it will be the standard It is preferable to provide a function for making the same thing into CAD and correcting the world coordinate system based on each other. The delivery of CAD data may be a communication means or a method performed via a storage medium such as FD or MO.

  A twenty-fourth means integrates CAD data generated and updated by a plurality of image information processing apparatuses in a visual information processing apparatus configured by applying the twenty-second means or a visual information processing system to which the twenty-second means is applied. Means is to enable CAD data generated and updated by a plurality of image information processing apparatuses to be integrated by another one or a plurality of image information processing apparatuses.

  The twenty-fourth means includes CAD data generated and updated by a plurality of image information processing apparatuses in a visual information processing apparatus configured by applying the twenty-second means or a visual information processing system to which the twenty-second means is applied. Means for integrating CAD data generated and updated by each of a plurality of image information processing devices with one or a plurality of other visual information processing devices dedicated for integration. An image information processing apparatus that processes information to generate CAD data only performs generation of CAD data and the like, and does not need to perform integration processing. Visual information processing devices that perform integration processing can concentrate on processing that integrates CAD data. It is possible to perform integration processing of CAD data efficiently at a high speed. When the number of visual information processing apparatuses that transmit CAD data increases, a plurality of visual information processing apparatuses that perform integration processing are used, and CAD data that is integrated to some extent is further integrated with another visual information processing apparatus. By doing so, it becomes possible to efficiently integrate the CAD data generated by photographing with more imaging devices. Of course, the integrated CAD data can be used as a three-dimensional map or the like.

  A twenty-fifth means integrates CAD data generated and updated by a plurality of image information processing apparatuses in a visual information processing apparatus configured by applying the twenty-second means or a visual information processing system to which the twenty-second means is applied. The means integrates the CAD data generated and updated by each of the plurality of image information processing apparatuses with another one or a plurality of image information processing apparatuses, and returns all or part of the integrated result to the image information processing apparatus. Is to do so.

  The twenty-fifth means includes CAD data generated and updated by a plurality of image information processing apparatuses in a visual information processing apparatus configured by applying the twenty-second means or a visual information processing system to which the twenty-second means is applied. Means for integrating CAD data generated and updated by each of the plurality of image information processing apparatuses (1) with another one or a plurality of image information processing apparatuses (2), and all or part of the integrated result. Is returned to the image information processing apparatus (1), so that the integrated CAD data can be used also by the image information processing apparatus at the end of the imaging apparatus. In other words, the integration result can be used without performing the integration processing by the image information processing apparatus itself. That is, even when the imaging apparatus moves for the first time to a place where the imaging apparatus has not been moved and CAD data has not been generated, it is possible to acquire and refer to the CAD data that has already been generated. . The same applies to the twenty-third means.

  The twenty-sixth means includes means for exchanging generated and updated CAD data in a visual information processing apparatus configured by selectively applying the twenty-second to twenty-fifth means or a visual information processing system to which the visual information processing apparatus is applied. And an image information processing function for integrating CAD data acquired from other visual information processing means.

  In such a twenty-sixth means, in the visual information processing apparatus configured by selectively applying the twenty-second to twenty-fifth means or in the visual information processing system to which the visual information processing system is applied, a plurality of image information processing apparatuses are respectively provided. In order to integrate the generated and updated CAD data, if a plurality of image information processing devices exchanges CAD data with other visual information processing devices by wireless communication, the mobility of each imaging device is improved. As a result, it is possible to move over a wide range without being restricted by a communication cable or the like. Further, real-time CAD data can be exchanged without using a storage medium.

  The twenty-seventh means is a visual information processing apparatus and visual information processing system configured by selectively applying the first to twenty-sixth means, wherein the image information processing apparatus is a position or orientation of a moving imaging device or its It is to have a function for obtaining both pieces of information and a function for outputting the obtained position information and / or posture information.

  The twenty-seventh means applies the visual information processing apparatus configured by selectively applying the first to twenty-sixth visual information, the visual information processing apparatus having CAD data in advance, or those visual information processing apparatuses. In the visual information processing system, the image information processing apparatus has a function of obtaining information on the position and / or orientation of the moving imaging device and a function of outputting the obtained position information and / or orientation information. The position and orientation information of the apparatus can be used on the application system side such as another CPU. When applied as a visual guidance control device for an automatic machine such as a robot, the relative attachment position relationship between the imaging device and the robot hand is known by attaching the imaging device to the tip of a robot hand or the like accessing the target object. Therefore, if the position and orientation of the imaging device are known, the position and orientation of the robot hand can be known, and the robot hand can be guided and controlled to an arbitrary target point in the movement space of the generated CAD data.

  According to a twenty-eighth means, in the visual information processing system configured by applying the twenty-seventh means, information on the position and / or posture of the imaging apparatus that moves at a predetermined timing of a command signal input from the outside is output. It is to have a function.

  In such a twenty-eighth means, in the visual information processing apparatus or visual information processing system configured by applying the twenty-seventh means, a command signal from the outside is input and moved at a predetermined timing of the command signal. Since it has a function of outputting information on the position and / or orientation of the imaging device, it can request the information on the position and orientation of the imaging device at a timing required on the application system side such as another CPU. A system can be easily constructed.

  The twenty-ninth means is a visual information processing apparatus configured by selectively applying the first to twenty-eighth means and a visual information processing apparatus having CAD data in advance or their visual information processing apparatus. In the visual information processing system, the image information processing apparatus has a function of searching for CAD data based on a search command signal for CAD data and outputting information of the searched CAD data based on the command signal. That is.

  Such a twenty-ninth means includes a visual information processing apparatus configured by selectively applying the first to twenty-eighth means, a visual information processing apparatus having CAD data in advance, or a visual information processing apparatus thereof. In the visual information processing system to which is applied, the image information processing apparatus has a function of searching for CAD data based on the CAD signal search command signal and a function of outputting information of the searched CAD data. Therefore, information required on the application system side such as another CPU, for example, CAD data of a target object to be searched is searched, and information on the position and orientation is searched as CAD data information of the searched target object. For example, on the application system side, for example, the position and figure of the robot hand holding the target object It is possible to induce controlled. Furthermore, if the CAD data of the robot hand is periodically searched and the position and orientation of the searched robot hand CAD data is obtained as a search result, the information is fed back to the position and orientation of the robot hand. It can also be used for control, and it is possible to perform guidance control that corrects the deflection of the arm portion of the robot hand or the positional deviation of the base of the robot hand. In addition, since the CAD data of the target object to be searched is periodically searched, even when the target object is moving, the position and orientation information of the target object can be obtained in real time. It is possible to guide and control the movement of the target object to be moved by accessing the robot hand.

  The thirtieth means is a visual information processing apparatus configured by applying the twenty-ninth means or a visual information processing system to which the thirty-first means is applied. It is to search with CAD data or to have a search function for both.

  In such a thirty-third means, in the visual information processing apparatus configured by applying the twenty-ninth means or the visual information processing system to which the thirty-second means is applied, the function of searching for CAD data is to search using basic graphics, Searching with the CAD data read out for searching or having both search functions, it is possible to easily search various data such as CAD data of complicated shapes other than basic figures such as spheres, cylinders and cubes. become able to. For example, there are few basic figures that match the complex shapes in the real environment. If you keep the CAD data generated by the CAD data generation system and search for the same CAD data, you can use various shapes. That is, the CAD data can be searched. Of course, if it is the CAD data of the part read for exploration, it matches with high precision. If it has moved, it is also possible to recognize the amount of movement based on the search result.

  The thirty-first means is a visual information processing apparatus configured by applying the thirty-first means or a visual information processing system to which the thirty-first means is applied. CAD data to be read for search is previously photographed from various angles for each target object. It is to make it generate.

  In such a thirty-first means, in the visual information processing device configured by applying the thirty-first means or the visual information processing system to which the thirty-first means is applied, various CAD data to be read for searching are previously stored for each target object. Since it is generated by shooting from an appropriate angle, CAD data of an object having a complicated shape can be prepared in advance as in the case of the basic figure. Since CAD data can be generated by photographing from various angles, complete CAD data can be prepared in advance over 360 °. The CAD data read out first may be registered in the memory with a name (type ID) corresponding to a desk for a desk and a name (type ID) corresponding to a chair for a chair. In this way, for example, if CAD data of various objects is registered in a memory before supplying an apparatus (product), a user who uses the product reads out without generating CAD data even once. Even if the processing is performed, it is possible to use a lot of CAD data having a complicated shape for searching.

  According to a thirty-second means, in a visual information processing apparatus configured by applying the thirty-first means or a visual information processing system to which the thirty-first means is applied, a place where the target object is to be photographed is a predetermined target object only including the target object Is to be a room.

  In the thirty-second means, in the visual information processing apparatus configured by applying the thirty-first means or the visual information processing system to which the thirty-first means is applied, the target object is photographed only at the target object that is the photographing target. Since there is no other obstacle, the CAD data of only the target object can be easily generated.

  According to a thirty-third means, in the visual information processing apparatus configured by applying the thirty-th means or the visual information processing system to which the thirty-third means is applied, the CAD data read for search is obtained by cutting out a designated area. is there.

  In such a thirty-third means, in the visual information processing apparatus configured by applying the thirty means or the visual information processing system to which the thirty-third means is applied, the CAD data read out for searching can cut out the designated area. For example, only a signboard part of a specific building may be used. If the shape of a building is characteristic, such as a dome, CAD data in the minimum necessary range for each target to be searched so that a place including the entire building is read out. Can be read out for searching.

  In a thirty-fourth means, in the visual information processing apparatus configured by applying the thirtyth means or the visual information processing system to which the thirty-third means is applied, the CAD data read for searching is the basic figure or the CAD data read for searching. And a region matched based on a predetermined threshold or the region and its surrounding region are cut out.

  In such a thirty-fourth means, in the visual information processing apparatus configured by applying the thirtyth means or the visual information processing system to which the thirty-third means is applied, CAD data read out for searching is read out for basic graphics or searching. Since the region matched with the CAD data based on a predetermined threshold or the region and its surrounding region is cut out, the shape close to the basic figure or the shape close to the shape read for search is used. It is possible to easily limit the newly read range. For example, if you want to read all rectangular signboards for search using generated CAD data, set a threshold value for matching to a certain extent at first, and it will be close to the basic figure of the rectangular parallelepiped. Since several (rectangular signboard candidates) are read out, it is possible to easily select a desired one from among them.

  The thirty-fifth means is a visual information processing apparatus configured by selectively applying the thirtieth to thirty-fourth means or a visual information processing system to which the thirty-fourth means is applied, and features of geometric information of CAD data read for search It is to extract the feature amount, the feature amount of the color information, or both.

  Such a thirty-fifth means is the geometry of CAD data read out for searching in a visual information processing apparatus configured by selectively applying the thirtieth to thirty-fourth means or a visual information processing system to which the visual information processing system is applied. Since it is possible to extract the feature quantity of information, the feature quantity of color information, or both, the feature quantity data for the case of searching using the feature quantity next time Can be automatically created in a short time.

  The thirty-sixth means is a visual information processing apparatus configured by selectively applying the twenty-ninth to thirty-fifth means, or a visual information processing system to which the visual information processing system is applied. In other words, the feature amount of the color information or both feature amounts are used.

  Such a thirty-sixth means is a visual information processing apparatus configured by selectively applying the twenty-ninth to thirty-fifth means, or a visual information processing system to which the visual information processing system is applied. Since the feature amount, the feature amount of the color information, or both feature amounts can be used, even when the environment to be searched is wide and there is a lot of CAD data, the search can be efficiently performed in a short time. it can. That is, in a vast space composed of a lot of CAD data, the search CAD data and the posture and size are changed and matching processing is repeated many times to obtain the correlation coefficient (coincidence) and the closest one is obtained. The search process takes an enormous amount of time, but first narrows down the target range by geometric features such as size, number of irregularities, and aspect ratio, and then tries to search by shape matching or pattern matching. By narrowing down things, you can further narrow down the search range by using features such as reddish cups and reddish apples with red as the main component and colors that have a high overall ratio. Therefore, it is possible to reduce the number of processings for shape matching or pattern matching that requires time, and the search time itself can be shortened.

  According to a thirty-seventh aspect, in the visual information processing apparatus configured by selectively applying the twenty-ninth to thirty-sixth means or the visual information processing system to which the visual information processing apparatus is applied, the eigenvalue space matching method is used for the search. It is to be.

  As described above, the thirty-seventh means uses the eigenvalue space matching method for the search in the visual information processing apparatus configured by selectively applying the twenty-ninth to thirty-sixth means or the visual information processing system using the visual information processing apparatus. Therefore, even when shape matching is performed, it is possible to efficiently narrow down target candidates in a short time.

  The thirty-eighth means uses a three-dimensional potential matching method for searching in a visual information processing apparatus configured by selectively applying the twenty-ninth to thirty-seventh means or a visual information processing system using the same. Is to do so.

  The thirty-eighth means is a visual information processing apparatus configured by selectively applying the twenty-ninth to thirty-seventh means or a visual information processing system to which the thirty-seventh to thirty-seventh means are applied. Therefore, it is possible to efficiently perform final position-posture matching, that is, to accurately obtain the position and posture in the CAD data of a search target in a short time.

  A thirty-ninth means is a visual information processing apparatus configured by selectively applying the thirty-ninth to thirty-eighth means or a visual information processing system to which the thirty-ninth to thirty-eighth means are applied, and has a similar shape with a variable search time scale factor. To search for.

  Such a thirty-ninth means is a visual information processing apparatus configured by selectively applying the twenty-ninth to thirty-eighth means or a visual information processing system to which the thirty-ninth to thirty-eighth means are applied. Since a search is made for items that are close to each other, it is possible to eliminate a large amount of search data for searching for items that have the same shape and various sizes.

  The 40th means can set a search range at the time of search to a predetermined range in a visual information processing apparatus configured by selectively applying the 29th to 39th means or a visual information processing system to which the visual information processing system is applied. Is to make it.

  Such a 40th means is a visual information processing apparatus configured by selectively applying the 29th to 39th means or a visual information processing system to which the visual information processing apparatus is applied. Therefore, when the place to be searched for is already known to some extent by address, coordinates, etc., it is possible to efficiently search by searching only the range.

  The forty-first means is a visual information processing apparatus configured by selectively applying the twenty-ninth to forty-first means, or a visual information processing system to which the visual information processing apparatus is applied. The range of the field of view or the range expanded or reduced from the range of the field of view.

  Such a 41st means is inputted with a search range at the time of search in a visual information processing apparatus configured by selectively applying the 29th to 40th means or a visual information processing system to which the visual information processing system is applied. The range of the field of view of the image information or a range expanded or reduced from the range of the field of view, so in online search that searches in real time while generating CAD data, more accurate and reliable data while generating more detailed as-built data Search can be performed.

  The forty-second means is a visual information processing apparatus configured by selectively applying the twenty-ninth to forty-first means, or a visual information processing system to which the visual information processing apparatus is applied. Both or any of the output I / Fs of the searched CAD data information is to have a plurality of I / Fs.

  Such a forty-second means is a visual information processing apparatus configured by selectively applying the twenty-ninth to forty-first means, or a visual information processing system to which the visual information processing apparatus is applied. / F, both or any of the searched CAD data information output I / Fs have a plurality of I / Fs, so that even when many searches are desired at the same time, it is possible to efficiently give a search instruction and obtain a search result efficiently. become able to. That is, it is possible to easily improve the environment recognition speed and the search speed of many objects.

  In a visual information processing apparatus configured by selectively applying the 29th to 42nd means or in a visual information processing system using the same, the forty-third means is a basic graphic or CAD data for search and a predetermined threshold value. The part which matched based on this is managed with a specific name (ID).

  Such a 43rd means includes a basic graphic or search CAD data in a visual information processing apparatus configured by selectively applying the 29th to 42nd means or a visual information processing system using the visual information processing apparatus. Since the part matched based on a predetermined threshold value is managed with a specific name (ID), for example, if the coordinates and the address that existed together with the name (ID) are registered, when searching again, Since it is possible to perform a process of looking at the vicinity of the approximate coordinates and address directly from the name (ID), the search time can be greatly shortened. In addition, a part that is not managed with a name (ID) is an unrecognized part. Therefore, when an unrecognized part is newly recognized and registered, an unrecognized range is easily extracted. It becomes possible to do.

  In a visual information processing apparatus configured by selectively applying the 29th to 43rd means or a visual information processing system to which the 44th means is selectively applied, the basic figure or the search CAD data and a predetermined threshold value are used. In other words, a predetermined portion of the CAD data excluding the matched portion is managed with a predetermined name (ID).

  Such a forty-fourth means is a visual information processing apparatus configured by selectively applying the twenty-ninth to the forty-third means or a visual information processing system to which the visual information processing apparatus is applied, and a predetermined graphic and search CAD data. Since a predetermined portion of CAD data excluding a portion matched based on the threshold value is managed with a predetermined name (ID), basic figures and other searches that have been prepared so far have been prepared. It is possible to register a new block of CAD data not included in the CAD data for use by defining a new name. Of course, it may be added as search CAD data. A new name (ID) may be entered by a person, or if the system automatically defines the ID with a series of numbers, for example, the number is automatically assigned. It may be automatically provided.

  The forty-fifth means is a visual information processing apparatus configured by selectively applying the twenty-ninth to forty-fourth means or a visual information processing system to which the visual information processing apparatus is applied, and manages with a specific name (ID). In other words, human language is associated with the CAD data.

  The forty-fifth means provides a specific name (ID) in a visual information processing apparatus configured by selectively applying the twenty-ninth to forty-fourth means or a visual information processing system to which the visual information processing system is applied. Since human words are associated with the CAD data to be managed, a man-machine I / F with a human can be easily constructed. For example, if an object with name (ID) = 71 is registered so as to correspond to the name of a human word such as “red cup on the desk” or “cup”, when searching for a cup on the desk If instead of inputting name (ID) = 71, “red cup on the desk” is input, if “red cup on desk” is associated with ID = 71, human words are A search instruction can be easily issued using the ID.

  The forty-sixth means is that a visual information processing apparatus configured by selectively applying the twenty-ninth to forty-fifth means or a visual information processing system to which the visual information processing apparatus is applied uses knowledge data during search. is there.

  The forty-sixth means uses knowledge data at the time of searching in a visual information processing apparatus configured by selectively applying the twenty-ninth to forty-fifth means or a visual information processing system to which the visual information processing system is applied. Therefore, instead of searching in the dark clouds, first, the space to search for the target object can be narrowed down using the knowledge data, so that the search can be performed efficiently. That is, for example, a cup is assigned IDs 71, 72, 73, etc. and 70s, and a desk is provided with IDs 81, 82, 83, etc., 80s. If there is a description such as “...” and “desk” in the 80's ID, and there is a description “... is often above”. When searching for..., First, the CAD data in contact with the above is searched, and “Cup (ID = 70's) is the desk (ID = 80's). Assuming that knowledge data is often registered, the system can first search from CAD data on the desk when searching for a cup. You can postpone searching for things like under the desk or on the floor. , Cup when there on top of the desk, it is possible to end quickly the search. If it is not on the desk, the cup (ID = 70's) may be on the desk (ID = 80's) so as to search on the next most likely floor. It is possible to search for the next floor on the basis of such knowledge data when it is not on the desk. The knowledge data may be set in a priority order in more stages so that more knowledge data can be registered for the same type of object. In addition, with knowledge data such as “a large object standing on the ground (volume OO or more) is often a building”, a large object on the ground (a block of CAD data exceeding a predetermined volume) is first selected. May use knowledge data to narrow down to search candidates.

  In a visual information processing apparatus configured by applying the 46th means or a visual information processing system to which the 47th means is applied, knowledge data is input together with a name (ID) given to an object. It is to enable input.

  In such a 47th means, in the visual information processing apparatus configured by applying the 46th means or the visual information processing system to which the 46th means is applied, the input of knowledge data is performed when a name (ID) is given to an object. In this way, knowledge data can be input efficiently. For example, when the system registers a new name (ID) for an object, when it asks "What is this", the name is "cops", and as other knowledge data, If you want to distinguish whether it is something that can be grabbed by the robot on the system side or otherwise, you can hold it with the robot hand. You may make it input the knowledge data such as. In this case, when the human language is associated with the name (ID) of the object and is input offline, knowledge data related thereto may be input. By doing so, input omissions and input forgets are eliminated.

  Forty-eighth means, in the visual information processing apparatus configured by applying the above-mentioned 46th and 47th means or the visual information processing system to which the above-mentioned means is applied, can input knowledge data separately as input. Is to do.

  The forty-eighth means is a visual information processing apparatus configured by applying the forty-sixth or forty-seventh means or a visual information processing system to which the forty-eighth means is applied. When the device (product) is supplied to the user for the first time, for example, in a home robot, if the objects that can be used as the environment in the home are registered in advance, the user registers each one There is no need to go. If the pre-registered data is not only basic shape data for search but also related knowledge data, the environment recognition can be started immediately after handing it to the user. For this purpose, if knowledge data can be input collectively from a file or the like at the time of manufacture, knowledge data can be efficiently input before shipping. Of course, the user may select a file of the type of knowledge data that is required, and input from the file after purchase.

  The forty-ninth means is a visual information processing apparatus configured by selectively applying the forty-sixth to forty-eighth means or a visual information processing system to which it is applied, and knowledge data is input as written in human language. Is to be able to do it.

  Such a 49th means is a visual information processing apparatus configured by selectively applying the 46th to 48th means or a visual information processing system to which the visual information processing system is applied. Knowledge data is described in human language. Since it is possible to input as it is, it is easy to input and create knowledge data. This is because, as already described above, the knowledge data is “the cup (ID = 70s) is often on the desk (ID = 80s)” or the cup is first. (ID = 70's), if the desk is registered as (ID = 80's), this means that it is possible to express "the cup is often on the desk". Because it is a human language, a person can efficiently create knowledge data.

  In a visual information processing apparatus configured by selectively applying the 29th to 49th means or a visual information processing system to which the 50th means is selectively applied, the basic graphic or the search CAD data and a predetermined threshold value are used. The matching portion is replaced with basic graphics or search CAD data.

  Such a 50th means is a visual information processing apparatus configured by selectively applying the 29th to 49th means or a visual information processing system to which the visual information processing apparatus is applied, and a predetermined figure or CAD data for search and a predetermined information. Since the matching portion based on the threshold value is replaced with the basic figure or the CAD data for search, the amount of CAD data can be reduced to save the memory capacity. That is, for example, a rectangular parallelepiped that is defined by vertical, horizontal, and height dimensions of a rectangular parallelepiped composed of a total of six rectangular planes consisting of four lines is equivalent to a predetermined posture at a predetermined position. In this case, the data of 12 lines and 6 faces can be deleted by replacing the basic figure with a rectangular parallelepiped, which saves memory. If the part once read out as the search CAD data is replaced only with information such as the name (ID) of the read shape, its position and orientation, its size and color, the read out CAD data contains a lot of line data, It is possible to save the memory capacity of such data. When searching around the replaced environment or updating the CAD data, it may be replaced again with the CAD data for searching if necessary. Since it can be replaced, memory capacity is saved.

  A fifty-first means obtains movement information of an imaging device based on image information in a visual information processing apparatus configured by selectively applying the first to fifty means or a visual information processing system to which the visual information processing system is applied. When it is possible, the input image information is processed to obtain the position and orientation information of the imaging device. When the movement information of the imaging device cannot be obtained based on the image information, the imaging information is obtained based on the movement information signal of the imaging device. A function to obtain the position and orientation information of the apparatus and use it for CAD data generation is provided.

  Such a 51st means is a visual information processing apparatus configured by selectively applying the first to 50th means or a visual information processing system to which the visual information processing apparatus is applied. When the information can be obtained, the input image information is processed to obtain the position and orientation information of the imaging device, and when the movement information of the imaging device cannot be obtained based on the image information, the movement information signal of the imaging device is used. Since it has a function of automatically switching so as to obtain the position and orientation information of the imaging device, when the image information is insufficient and the movement information cannot be calculated, or the moving speed of the imaging device is fast, it is input from the imaging device. The rate of change due to movement of image information exceeds the image processing speed, and processing continues even if feature points on the image information are lost due to tracking tracking by image processing, etc. It is possible to continue on. For example, when the imaging device moves at a high speed, it is impossible to obtain the position and orientation of the imaging device by losing sight of the tracking-following feature point, and the amount of movement at that time can be detected by an imaging device movement sensor. Detecting based on the imaging device movement control information, the amount of movement that was lost when returning to low speed movement again is obtained from the information detected based on the imaging device movement sensor and the movement control information, and the position of the imaging device By correctly correcting the posture information, it is possible to know where the imaging device is located and looking in which direction at the time of returning to low speed movement, and therefore by matching with CAD data of the already generated moving space Correction is also possible, and tracking tracking can be resumed from that location. When an acceleration sensor or a gyro sensor is used as an imaging device movement sensor provided in the imaging device, since these sensors have sufficient response to high speed movement, it is possible to accurately detect the movement amount during high speed movement. . On the other hand, the acceleration sensor and the gyro sensor have a large error for long-time accumulated detection, but the accumulated error can be canceled by detecting the position and orientation by image processing during low-speed movement. A sensor or a gyro sensor can be effectively used as another sensor.

  A 52nd means is a visual information processing apparatus configured by selectively applying the 1st to 51st means or a visual information processing system to which the visual information processing apparatus is applied, and moving to move the imaging apparatus It is to have a function of judging whether the target object in the image information is moving or stopped with reference to the body movement information or movement control information.

  In such a 52nd means, in the visual information processing apparatus configured by selectively applying the first to 51st means or in the visual information processing system to which the visual information processing system is applied, it is determined whether or not the imaging apparatus is moving. When an object to be imaged that moves to the full field of view of the imaging device is imaged, it has a function to determine whether the object to be imaged is moving or stopped by the information signal. In addition, it is possible to easily distinguish whether the object to be imaged is moving or whether the imaging apparatus itself is moving.

  In a visual information processing apparatus configured by selectively applying the first to 52nd means or a visual information processing system to which the 53rd means is selectively applied, halation in the image information obtained by the imaging apparatus is obtained. If there is a halation in the image information, the area of the halation or the entire area of the image information is set as an unprocessed area.

  Such a 53rd means is a visual information processing apparatus configured by selectively applying the above 1st to 52nd means or a visual information processing system to which the visual information processing apparatus is applied. If there is a halation in the image information, the halation area or the entire area of the image information is set as an unprocessed area. Recognition and the like can be easily prevented.

  In a visual information processing apparatus configured by selectively applying the first to 53rd means, or in a visual information processing system to which the 54th means is applied, halation in image information obtained by the imaging apparatus is obtained. Even if it is detected, if the area is small and the coordinates of the halation part can be calculated with a predetermined accuracy, even if there is a halation in the image information, the part of the halation is processed. is there.

  Such a 54th means includes a visual information processing apparatus configured by selectively applying the first to 53rd means or a visual information processing system to which the visual information processing apparatus is applied. If the area is small and the coordinates of the halation part can be calculated with a predetermined accuracy even if the halation is detected, the part of the halation is processed even if there is halation in the image information. Therefore, it becomes possible to easily convert the glitter of the water surface, diamond dust (ice crystal), sun pillar, etc. into CAD data. At this time, of course, when a needle-like portion that appears like a light line is generated by halation, the needle-like portion can be ignored as noise and only the center portion can be processed.

  According to a 55th means, in the visual information processing apparatus configured by selectively applying the first to 54th means or the visual information processing system to which the visual information processing system is applied, the coordinate value of the CAD data has a unit of length. Is to do so.

  Such a 55th means is a visual information processing apparatus configured by selectively applying the first to 54th means or a visual information processing system to which the visual information processing apparatus is applied. Since the unit is set, it is not necessary to manage the absolute length on the application side.

  The fifty-sixth means is a visual information processing apparatus constructed by applying the fifty-fifth means or a visual information processing system to which the fifty-fifth means is applied, and at least one CAD is provided as means for giving the coordinate value of the CAD data a unit of length. The absolute value dimension of the data and its CAD data is input.

  The fifty-sixth means is a visual information processing apparatus configured by applying the fifty-fifth means or a visual information processing system to which the fifty-fifth means is applied. Since at least one CAD data and the absolute value dimension of the CAD data are input, the absolute length can be easily defined.

  In a visual information processing apparatus configured by applying the 55th means or a visual information processing system to which the 57th means is applied, the 57th means has at least two means as means for giving the coordinate value of CAD data a unit of length. The absolute value distance between the CAD data and the CAD data is input.

  Such a 57th means is a visual information processing apparatus configured by applying the 55th means or a visual information processing system to which the visual information processing apparatus is applied, as means for giving a unit of length to the coordinate value of CAD data, Since at least two CAD data and the absolute value distance between the CAD data are input, a reference object, for example, a ruler, is photographed, and the interval between the scales is read in association with the absolute length. And the absolute length can be easily defined.

  A 58th means is a visual information processing apparatus configured by selectively applying the above first to 57th means or a visual information processing system to which the visual information processing apparatus is applied. It is supposed to be a thing.

  In such a 58th means, the visual information processing apparatus constructed by selectively applying the first to 57th means or the visual information processing system to which the visual information processing system is applied is entirely or partially implemented as an LSI chip. Therefore, it is possible to easily provide a visual information processing apparatus capable of high-speed processing at low cost by mass production of LSI chips, and it becomes easy to construct an application system combined with another CPU or the like.

  According to a 59th means, a visual information processing apparatus configured by selectively applying the first to 57th means or a visual information processing system to which the visual information processing system is applied realizes an image information processing function by software information processing. It is supposed to be configured as described above.

  As described above, the 59th means is realized by information processing by software in a visual information processing apparatus configured by selectively applying the first to 57th means or a visual information processing system to which the visual information processing system is applied. For example, when a general-purpose personal computer is used for the mobile robot control device, the software for the visual information processing device is added to the same personal computer together with the control software for the mobile robot. It becomes possible to construct an application system without adding.

  In a visual information processing apparatus configured by selectively applying the first to 59th means or a visual information processing system to which the 60th means is selectively applied, image information obtained by a plurality of imaging devices is predetermined. It is to switch the timing and input to the same image information processing apparatus.

  Such a 60th means is a visual information processing apparatus configured by selectively applying the first to 59th means or a visual information processing system to which the visual information processing apparatus is applied, and image information obtained by a plurality of imaging devices. Are switched at a predetermined timing and input to the same image information processing apparatus or image information processing system. For example, the image information of a plurality of imaging devices is switched instantaneously in the order in which the imaging devices have moved. By doing so, it is possible to input image information corresponding to the moment when the image-capturing device has moved instantaneously, and it is possible to obtain a system that can easily cope with a target object that moves at high speed. In this case, the imaging device is not limited to a moving imaging device, and may be a fixed imaging device. In the case of an imaging device mounted on a pan / tilt mechanism with a fixed imaging device, pan / tilt angle information is input to the system so that the position and orientation information of the imaging device is known and more efficiently processed. can do.

  A 61st means includes a visual information processing apparatus configured by selectively applying the first to 60th means to a control device of a robot for positioning a specific part, or a visual information processing system using the visual information processing system. Is to apply.

  Such a 61st means is a visual information processing apparatus configured by selectively applying the first to 60th means to a control device in a robot for positioning a specific part, or visual information using the visual information processing apparatus. Since the processing system is applied, for example, in a remote control robot, a system for automatically positioning the hand can be easily constructed. Specifically, for example, if the imaging device is attached to the hand and the position and orientation information of the imaging device is obtained, the position of the hand can be corrected and controlled to a target position based on the information. Further, if the target object to be accessed by the hand is searched from the CAD data and the position and orientation information of the target object is obtained, the position and orientation of both the imaging device and the target object can be known, so that more accurate It is possible to perform guidance control so that the hand is accessed to the target object. In addition, when an image pickup device cannot be attached to the hand, a separate image pickup device for photographing the hand is provided, and the CAD data of the hand is searched using the image information of the image pickup device to directly determine the position and posture of the hand. It can also be detected and used for control.

  A 62nd means is to apply a visual information processing apparatus configured by selectively applying the first to 60th means or a visual information processing system to which the visual information processing system is applied to a control apparatus for a mobile robot.

  Such a 62nd means applies a visual information processing apparatus configured by selectively applying the first to 60th means to a control device for a mobile robot or a visual information processing system to which the visual information processing system is applied. Therefore, it becomes possible to easily perform positioning and posture control of the mobile robot itself. For example, if an imaging device is mounted on a mobile robot, the position and orientation information of the imaging device indicates information on the position and orientation of the mobile robot. Therefore, the position and orientation of the mobile robot can be controlled based on that information. Thus, positioning control, attitude control, guidance control, etc. of the mobile robot can be easily performed. If an imaging device is mounted on a ROV or the like that floats in water and this device is applied, even if it is located deep in the water and the ROV in the water cannot be seen directly from above the water surface, for example, the CAD of the surrounding environment If the data and the position and orientation of the imaging device are displayed on the display device of the console, remote operation of the underwater ROV can be easily performed while confirming the position and orientation with the display device. It becomes like this. In addition, if the final target position and posture are input so as to be positioned in front of an arbitrary place on the CAD data and the underwater ROV is automatically controlled to be the target position and posture, the automatic operation can be performed. A control system can also be easily constructed. Of course, an imaging device can be mounted on an airship or an airplane and applied to remote operation and automatic control of a flying robot.

  A 63rd means is to apply a visual information processing apparatus configured by selectively applying the first to 60th means or a visual information processing system to which the first to 60th means is applied to a processing device of a character reading device. .

  The 63rd means applies a visual information processing apparatus configured by selectively applying the first to 60th means or a visual information processing system using the same to the processing device of the character reading apparatus. Therefore, for example, a system that makes it easy to recognize predetermined characters by matching processing of three-dimensional CAD data even for characters that are difficult to read by two-dimensional image processing, such as imprinted characters with three-dimensional unevenness, is easily constructed. can do. Also, since characters that change in time series can also be generated in time series, the movement pattern is known in advance, for example, characters displayed to flow on an electric bulletin board or signboards swaying in the wind. It is possible to construct a system that can easily read characters that do not move.

  A sixty-fourth means is that a visual information processing apparatus configured by selectively applying the first to sixty means or a visual information processing system to which the visual information processing system is applied is applied to a processing device of a three-dimensional measurement apparatus. is there.

  As such a 64th means, a visual information processing apparatus configured by selectively applying the first to 60th means or a visual information processing system using the same is applied to a processing apparatus of a three-dimensional measuring apparatus. Therefore, it is possible to easily realize a 3D measuring device necessary for field surveys of plants, car design design, CG characters, and the like. That is, even if a general-purpose imaging device is used to generate a CAD data by capturing an image of the imaging device close to the target object without using a system using an imaging device with high-resolution resolution, Since CAD data with high accuracy can be generated, it is possible to generate CAD data with high accuracy based on image information obtained by photographing with a general-purpose imaging apparatus, and to refer to the CAD data. For example, a highly accurate 3D measurement system can be easily constructed using a general-purpose imaging device.

  A 65th means enables a three-dimensional measuring apparatus constructed by applying the 64th means or a processing apparatus thereof to convert the generated or updated CAD data into a predetermined CAD data format. That is.

  In the three-dimensional measuring apparatus constructed by applying the 64th means or its processing apparatus, the 65th means as described above, when taking out CAD data generated or updated, for example, a predetermined CAD data format, Since it can be converted into a DXF format or the like, CAD data generated in an as-built format can be easily used in a general-purpose CAD system such as Auto-CAD, for example. Here, the conversion may be performed at the time of output, or the conversion may be performed on the internal memory so that it can be output separately.

  A 66th means is a three-dimensional measuring apparatus constructed by applying the 64th and 65th means or a processing apparatus thereof, and matches a basic figure below a predetermined threshold when CAD data generated or updated is taken out. It is to replace the part with a basic figure.

  Such a 66th means is a three-dimensional measuring apparatus configured by applying the 64th or 65th means or its processing apparatus, when CAD data generated or updated is taken out below a predetermined threshold value. Since the part that matches the basic figure is replaced with the basic figure, it is usually stored in a data format that is easy to use for updating the CAD data in real time. it can. Further, by replacing the generated CAD data with a basic graphic when the CAD data is taken out for use, the CAD data can be converted into data that can be easily used by other applications using the basic graphic. As the basic figure, a sphere, a cylinder, a rectangular parallelepiped, a cone, or the like is prepared, and the dimensions may be varied to perform matching. The remaining CAD data that does not match the basic figure can be converted as it is if the application can handle line data and surface data. In the case of an application other than the basic figure, the part that cannot be converted to the basic figure It is possible not to take out.

  A 67th means is a three-dimensional measuring device configured by selectively applying the 64th to 66th means or a plurality of imaging devices moved by the processing device, and flies in the sky with a traveling vehicle running on the ground. The imaging device is mounted on the flying vehicle.

  Such a 67th means includes a three-dimensional measuring device configured by selectively applying the 64th to 66th means or a plurality of imaging devices moved by the processing device, and a traveling vehicle running on the ground. By using an imaging device mounted on a flying object flying in the sky, a more accurate three-dimensional map of the townscape can be easily created. That is, the visual information processing device of a traveling vehicle traveling on the ground generates detailed CAD data of a range where a road or a house is photographed from below, but on the roof of the house or on a tree growing on the ground. If you are looking up from below, you cannot get enough image information, so it is not possible to generate enough CAD data at the top, but you can use the image information taken from the flying object in the sky. By integrating with the generated CAD data, it is possible to easily create an accurate three-dimensional map of the townscape with no generation omission from the bottom to the top.

  The 68th means is to apply a visual information processing apparatus configured by applying the first to 60th means to the processing apparatus or a visual information processing system to which the navigation apparatus is applied.

  The 68th means applies a visual information processing apparatus configured by selectively applying the first to 60th means to the processing apparatus or a visual information processing system using the same in the navigation apparatus. Therefore, it becomes possible to easily realize a navigation device that does not require GPS. That is, by connecting an imaging device to a moving navigation device and generating and updating CAD data of the surrounding environment by the imaging device, the position and orientation information of the imaging device is extracted in real time, and the environment CAD data and the imaging By displaying the position and orientation information of the apparatus, the system can easily confirm the current position. The environmental CAD data may be given in advance to the navigation device as a certain amount of CAD data. For example, only the two-dimensional map information is provided, and the three-dimensional CAD data generated as the vehicle runs is generated. A two-dimensional map and three-dimensional CAD data may be associated from a road or a building near an intersection. The CAD data to be created in advance may be input with a characteristic building or the like off-line, or the CAD data of the townscape generated in advance by an automobile or flying object is generated and synthesized by the visual information processing apparatus of the present invention. It is also possible to use the one that has been used. Further, the generated as-built CAD data may be exchanged with other navigation devices that have moved to generate CAD data, or may be generated by a plurality of navigation devices via a central information processing center or the like. CAD data may be exchanged. The attachment relationship between the imaging device and the moving body is known, and by setting the relationship in advance, the position and orientation of the moving body are displayed instead of displaying the position and orientation of the imaging device. You can also Also, if the destination in the environmental CAD data is input, what route should be taken to move to the destination from the destination in the CAD data and the current position of the moving body, etc. Navigation display may be configured and executed in the same manner as a GPS-equipped navigation device.

  A 69th means is a navigation apparatus constructed by applying the 68th means, one or a plurality of other navigation apparatuses different from one navigation apparatus, an imaging apparatus connected to the navigation apparatus, or Information on the position or orientation or position and orientation relating to the navigation device or a mobile body equipped with the navigation device can be taken into the one navigation device, or can be taken into mutual.

  Such a 69th means is a navigation device constructed by applying the 68th means, one or a plurality of other navigation devices (2) different from the navigation device (1) or the navigation device thereof. Since the position or orientation or position and orientation information relating to the moving body of the imaging device connected to (2) or the navigation device (2) can be taken into the navigation device (1), or can be taken into each other. In addition to the information on the own position and the surrounding environment, each navigation device can refer to the position information of other navigation devices (that is, other moving objects) located near the surroundings, and the blind spot of the imaging device. The danger that another moving body pops out from the shadow of the building and encounters the head Situation and can each other of the navigation device is detected in advance, Toka inform it in advance, so it is possible to prevent a collision with Toka brakes the movable body. In addition, if a similar navigation device is attached to children or elderly pedestrians, the navigation device installed in the car will detect pedestrians and warn the pedestrians in advance. This makes it possible to prevent accidents from occurring. The navigation device attached to the pedestrian may transmit identification information indicating that it is a pedestrian in addition to the position information thereof, or may be registered in advance at the central information processing center. In addition, the identification contents can be distinguished from the type of the moving body such as a car, a two-wheeled vehicle, or a pedestrian, and the response process at the time of abnormal approach can be performed in an optimum manner according to the type of the moving body Can be able to.

  The 70th means is to apply a visual information processing apparatus configured by selectively applying the first to 60th means or a visual information processing system to which the visual information processing system is applied to the control device of the picking system.

  In such a 70th means, a visual information processing apparatus configured by selectively applying the first to 60th means to a control device of a picking system or a visual information processing system using the same is applied. Therefore, it is possible to easily construct a picking system by preparing at least one ordinary general-purpose imaging device. That is, by attaching an ordinary general-purpose image pickup device to the tip of the manipulator to be handled and generating work CAD data based on image information from the image pickup device, the image is first taken from the periphery of the work, so that the work as-built CAD data can be generated. Here, the imaging apparatus may be configured to obtain the same image information as when the imaging apparatus is moved around the work by fixing a plurality of signals around the work and switching the image information signal. After the as-built CAD data of the work has been generated, the shape of one part in the work is registered in advance as CAD data for search, and the CAD data of the work in which many generated parts are stacked Find the position and orientation information of each part to be grasped by the control device of the manipulator so that the position and orientation information of each part is obtained by the search function and picked by the manipulator for handling in order from the outermost part. If the manipulator is controlled by giving it, a picking system can be easily constructed. In particular, when an imaging device is attached to the tip of the manipulator, the imaging device can accurately generate CAD data by photographing the workpiece by bringing the imaging device close to the workpiece. It is possible to generate CAD data with high accuracy even when shooting with a general-purpose general imaging apparatus instead of the apparatus. Even if the shape of a workpiece stacked with components removed by picking is changed, the CAD data of the workpiece is updated, so that picking control can always be handled according to the state at that time.

  A 71st means is applying the visual information processing apparatus comprised by selectively applying the said 1st-60th means to the processing apparatus of a test | inspection apparatus, or the visual information processing system which applied it.

  Such a 71st means applies a visual information processing apparatus configured by selectively applying the first to 60th means to the processing apparatus or a visual information processing system using the same in the inspection apparatus. Therefore, a three-dimensional inspection apparatus can be easily obtained. That is, for example, if an imaging device is provided at the tip of a manipulator and the like and the CAD data to be inspected is generated by imaging the region to be inspected from various angles with this imaging device, Since the three-dimensional unevenness is accurately reproduced, for example, if it is possible to extract the position and orientation information with a predetermined part such as whether or not the predetermined part is properly attached at a predetermined place, it is easy An automatic inspection device can be constructed.

  The 72nd means is that the inspection apparatus constituted by applying the 71st means or the imaging apparatus used for the processing apparatus is a microscope.

  In such a 72nd means, since the inspection apparatus configured by applying the 71st means or the imaging apparatus used for the processing apparatus is a microscope, the generated three-dimensional CAD data is fine 3 Since it becomes a dimensional uneven surface, for example, it is possible to realize an inspection apparatus that can easily and efficiently inspect fine scratches that cannot be seen unless the angle of a CD or DVD is changed.

  A 73rd means is applying the visual information processing apparatus comprised by applying the said 1st-60th means to the control apparatus of an autonomous control type robot, or the visual information processing system which applied it.

  Such a 73rd means includes a visual information processing apparatus configured by selectively applying the first to 60th means to a control device of an autonomous control robot, or a visual information processing system using the visual information processing system. Since it is applied, it is possible to easily construct an autonomous control robot. That is, if an imaging device is attached to a robot, it is possible to easily generate environmental CAD data by applying a visual information processing device, and the position and orientation information of the imaging device in the environment and the position and orientation of the robot itself. Information can also be obtained easily. Furthermore, since it is possible to search for specific CAD data in the environment, the presence / absence of CAD data prepared for searching in advance, and the position and orientation information of the CAD data, if any, can be obtained. If it is a target object, the robot itself can be controlled with respect to the work target object in any way by the position and posture of the work target object and the position and posture of itself. For example, when trying to execute a work content to be found and returned by searching for a specific work target object, the robot uses the navigation function to move the robot along the route planned by the navigation, and then the work target object. When the work target object is found, the robot itself is approached at a predetermined position and posture, the work target object is held by the robot, and the navigation function is again performed. Control to return to the start point using can be easily realized. If there is an obstacle on the way, the CAD data of the environment is generated and updated at any time, so it is easy to recognize this obstacle as an obstacle. For example, if you prepare a program that searches for an avoidance route while avoiding obstacles, you can autonomously execute the target work as planned even if there is an obstacle on the way. Become. Here, what is called an autonomous control type robot may be any of an automobile, an automatic guided vehicle, a manipulator, and a humanoid robot, but is a robot having a function of controlling the operation by the robot itself. Autonomous control robots may basically be programmed to follow human commands, but for example in the case of pet robots, etc., if charging is necessary, charge towards the charger. It may be programmed to give priority to doing and not necessarily follow the command. Also, when applied to watchdog robots, it is easy to build a system that recognizes suspicious clothes and suspicious movements of surrounding people and programs suspicious people to follow them. Can be realized.

  According to a 74th means, in the autonomous control type mobile robot constructed by applying the 73rd means or a control device thereof, the target position and / or posture, or the target position and / or posture are moved. When the related contents are instructed, the robot autonomously moves to the destination according to the instruction contents, responds that the movement is completed when the movement is completed, and responds when the robot determines that it is difficult to complete the movement. Is to do.

  Such a 74th means moves in a target position and / or posture or a target position and / or posture in an autonomous control type mobile robot constructed by applying the 73rd means or its control device. If the robot determines that it is difficult to finish, it will make it move to the destination autonomously according to the contents of the instruction, respond to the end when the movement is completed, Since a response to that effect is given, it is possible to easily obtain a robot or a car that automatically moves to the destination when the destination is designated. For example, when a destination is input by an automobile, a route to the destination is displayed by the function of the navigation device. Here, the automobile is automatically driven according to the displayed route. In the middle, the surrounding environment is generated as CAD data. Therefore, when an obstacle that is not on the map is newly detected, the obstacle avoidance program prepared for each obstacle pattern is handled in advance. If avoidance control is performed by a program, it is possible to continue automatic driving. As an obstacle avoidance program, the obstacle on the left front should avoid the right side, or if another car is on the front as an obstacle, make it recognize that it is a car from the shape. In the case of a car, it may be possible to prepare an avoidance program that can cope with various possible patterns such as waiting until it starts moving or decelerating. Here, when a pattern for which an avoidance program is not prepared is encountered, the system may determine that it is difficult to terminate and respond accordingly. If there is a traffic signal on the way, CAD data of the traffic signal is prepared in advance for searching, and the traffic signal is searched from the surrounding environment, and it corresponds to red, blue and yellow in the traffic signal. If you check the lighting status of the lamp, if it is red or yellow, make it stop and wait until blue lights at the signal, and if it is blue, program it to continue automatic driving while avoiding surrounding obstacles as it is In addition, automatic driving can be easily performed even on roads with traffic signals. When the car finally stops at a predetermined location according to the position and orientation of the destination, the system returns an end response, so that the passenger can easily know that the destination has been reached. If it is answered that it is difficult to continue, the automatic operation may be performed again after switching to the manual operation to avoid the failure. In addition, when a typical building or place on the travel route is recognized on the way, guidance can be made that the system is currently passing there. As described above, in automatic driving control, it is necessary to correctly recognize the road conditions and continue automatic driving control using an appropriate program corresponding to the conditions. It is necessary to recognize many objects including features such as shape and color, but by applying the visual information processing apparatus of the present invention, it is possible to easily recognize them as described above. Therefore, an autonomous mobile robot such as an automobile traveling autonomously as described above can be easily realized. Actually, in addition to visual information processing, information on surrounding sounds, for example, visual information on the direction in which the sound source is supposed to be detected by detecting the sound of a car traveling or horn with a directional microphone. As the processing is performed preferentially, the recognition speed of the surrounding environment may be more accurately performed at high speed using the result of sound information processing. This function can also be applied to an automatic driving of a car that uses a garage operation in an automatic operation by using it for a garage operation that is difficult even if the moving distance is short. In addition, it can be applied to an automatic cleaner so that a map of the first environment can be automatically generated, and can be managed using CAD data that generates a cleaned range and a range that has not yet been cleaned.

  In the autonomous control type working robot constructed by applying the 73rd means or its control device, the 75th means performs work autonomously according to the instruction content when the work content is instructed and It is to respond to the end, and when the robot determines that it is difficult to end, a response to that effect is made.

  According to such a 75th means, in the autonomous control type working robot constructed by applying the 73rd means or its control device, when the work content is designated, the work is autonomously performed according to the content of the work. If the robot determines that it is difficult to finish, it responds to that effect. Also in the robot system to be performed, it is possible to obtain an autonomous control type working robot that works autonomously. In other words, if the target work is, for example, a valve overhaul and inspection work, the movement of a series of work robots is organized as an operation program, control is executed in that order, and all parts of the valve are programmed. If it is accurately located at the planned location, the valve can be disassembled and inspected automatically with a simple program-controlled robot. The position of the valve varies from valve to valve, and it is actually assembled in various positional relationships depending on the situation where the same valve is assembled. Therefore, basically, a program for performing work in accordance with the above-described automatic program is executed. On the way, the target part handled by the robot is recognized by CAD data generated by as-built, and the position and orientation of the part are accurately determined. By detecting and controlling the robot, even if the position of the valve is greatly deviated, the valve can be easily disassembled and inspected. In addition, if a large amount of part shape data is prepared in advance as search CAD data so that various parts can be recognized, it is possible to cope with various types of valve disassembly and inspection. However, if an unrecognizable part comes out in the middle of disassembly, it becomes unrecognizable, and it is difficult to continue the work at that time. When the work has been completed to the end, it can be easily confirmed that the work has been completed by responding to that effect. As described above, since the visual information processing apparatus of the present invention can easily recognize an object, it is possible to easily realize an autonomously controlled work robot that can automatically perform such work. It becomes. If applied to a care robot for people who need care, it recognizes the position of the mouth of the person who needs care, such as a spoon or a cup, and takes the soup in the cup with a spoon to take care of the caregiver. The control to bring it to the mouth can be easily performed. In such a care robot, a system in which a caregiver performs a task that the caregiver wants the robot to perform can be easily configured by giving a general command to the robot.

  The 76th means is that an autonomous control type robot or a control device thereof configured by selectively applying the 73rd to 75th means, wherein the robot operates safely in an independent system from the autonomous control system. Is to be able to monitor.

  Such a 76th means is an autonomous control type robot or a control device configured by selectively applying the 73rd to 75th means, and the robot operates safely in an independent system from the autonomous control system. Therefore, even when the autonomous control type robot is moving in a remote place where the eyes cannot reach, the robot can be monitored safely. Since the independent system monitoring means is provided, it is possible to always monitor without being influenced by complicated autonomous control. The monitoring may be performed directly by a human or may be monitored using an automatic monitoring device.

  The 77th means is an independent system from the autonomous control system configured by applying the 76th means, and means for enabling monitoring that the robot is operating safely from the independent control system. It is supposed to be a surveillance camera system.

  According to the 77th means, the independent control system can be used to monitor that the robot is operating safely in an independent system from the autonomous control system configured by applying the 76th means. By using the surveillance camera system as described above, it is possible to perform direct monitoring by video, so that the situation at the site can be grasped more accurately.

  The 78th means is an autonomous control type robot or the control device configured by applying the above 75th and 76th means, and when an emergency response is required during safety monitoring of the robot, It is to be able to execute emergency response control of the robot in an independent system.

  The 78th means as described above is an autonomous control type robot or the control device configured by applying the 75th or 76th means, and when emergency response is required during safety monitoring of the robot, Since the emergency response control of the robot can be performed by an independent system from the autonomous control system, an emergency stop or switching to the manual can be surely performed in the event of an emergency. In addition, when the monitoring video system is interrupted, the robot may be provided with a function of automatically or manually terminating it.

  A 79th means includes various robots, a character reading device, a three-dimensional measuring device, a navigation device, a picking system, an inspection device, a control device, or a processing device, which are configured by selectively applying the 61st to 78th means. The man-machine interface part has a voice recognition function and / or a voice synthesis function.

  The 79th means includes various robots, character reading devices, three-dimensional measuring devices, navigation devices, picking systems, inspection devices, control devices, or processing devices constructed by applying the 61st to 78th means. However, since the man-machine interface part is provided with a voice recognition function and / or a voice synthesis function, I / F with a human is further facilitated. For example, when a command is received in voice recognition and it is correctly received, a response of “Yes” is returned. When execution of the command is finished, “Finished”, or when it becomes difficult to continue Since “cannot be executed” is responded to, there is a response from the robot's instruction and result, so that a human can easily understand the work result. Furthermore, the work situation in the middle may be output as a voice so that the situation can be easily understood by humans.

  According to an eighty-eighth means, the image pickup apparatus has a signal processing circuit which rotates a one-dimensional or two-dimensional image pickup device (CCD sensor), and can capture a CCD sensor signal and output it as image information in synchronization with the rotation The signal processing circuit is configured to rotate together with the CCD sensor.

  Such an 80th means is a signal processing circuit that allows a one-dimensional or two-dimensional image sensor (CCD sensor) to rotate in the image pickup apparatus, captures the signal of the CCD sensor in synchronization with the rotation, and outputs it as image information. Since this signal processing circuit is configured to rotate together with the CCD sensor, it is possible to easily realize an image pickup apparatus capable of taking a wide range of images. In addition, since the signal processing circuit is rotated together with the rotating CCD sensor, it is not necessary to exchange signals between the CCD sensor and the signal processing through a rotating part such as a slip ring, and a wide range is photographed with a rotary method. An image pickup apparatus that can do this can be readily realized.

  According to an 81st means, in the imaging apparatus constructed by applying the 80th means, the signal processing circuit can input a command signal, and can process and output image information based on the command content of the command signal. is there.

  In such an 81st means, in the imaging apparatus constructed by applying the 80th means, the signal processing circuit can input the command signal and process and output the image information based on the command content of the command signal. Therefore, it is possible to efficiently extract only the necessary part of information.

  The 82nd means is an imaging device constructed by applying the 80th and 81st means, and has wireless communication means for communication for exchanging signals and information with the outside, and the wireless communication means is a sensor. It is comprised so that it may rotate with.

  Such an 82th means is an imaging apparatus constructed by applying the 80th or 81st means, and has a wireless communication means for communication with the outside, and this wireless communication means rotates together with the CCD sensor. With this configuration, it is possible to eliminate the need for signal sliplin and the like for extracting signals and information such as video signals and control signals from the rotating part to the stationary part.

  The 83rd means is to provide an electric power generation device in an imaging apparatus configured by selectively applying the 80th to 82nd means, and to configure the electric power generation device to rotate together with the sensor.

  The 83rd means has an electric power generation device in the imaging apparatus configured by selectively applying the 80th to 82nd means, and the electric power generation device rotates with the CCD sensor. Since it comprises, the slip ring for power supply can be made unnecessary.

  An 84th means is a visual information processing apparatus configured by selectively applying the first to 60th means in an imaging apparatus by selectively applying the 80th to 83rd means, or an application thereof. This visual information processing system or visual information processing system is configured to rotate together with the sensor.

  Such an 84th means is a visual information processing system configured by selectively applying the first to 60th means in an imaging device configured by selectively applying the 80th to 83rd means. Apparatus or a visual information processing system to which the apparatus is applied, and this visual information processing apparatus or visual information processing system is configured to rotate together with a CCD sensor. A visual information processing apparatus can be configured easily.

  The 85th means integrates information sent from a plurality of terminals at a central information center, and provides the user with all or part of the integrated information. It is supposed to include data and terminal status information.

  Such an 85th means integrates information sent from a plurality of terminals in a central information processing center, and provides information in a comprehensive information service system that provides all or part of the integrated information to the user. By including CAD data or CAD data and terminal status information, CAD data generated by other people can be easily used.

  A 86th means is a visual information processing system configured by selectively applying the first to 60th means to a processing device of the system in the information integrated provision service system constituted by applying the 85th means. An image processing apparatus or a visual information processing system to which the apparatus is applied or an imaging apparatus configured by selectively applying the 80th to fourth means is applied.

  The 86th means is configured by selectively applying the first to 60th means to the processing device of the system in the information management service system configured by applying the 85th means. Since the visual information processing apparatus or the visual information processing system to which the visual information processing apparatus is applied or the imaging apparatus configured by selectively applying the 80th to 84th means is applied, it is easy to generate and integrate as-built CAD data. Is possible.

  In the monitoring service providing system that centrally monitors information sent from one or a plurality of terminals at the center, the 87th means searches for a user by a request from the user, tracks the user during monitoring, And continuously monitoring the user's surroundings.

  In the monitoring service providing system that centrally monitors information sent from one or a plurality of terminals at the center, the 87th means searches for a user by a request from the user and tracks the user during monitoring. Thus, since the user and the user's surroundings are continuously monitored, it is possible to easily construct a monitoring service system that guarantees the safety of the user. Here, tracking may be performed by a manual device, or automatic tracking may be performed by image processing or the like. In addition, users of autonomous control robots can use this service for robot monitoring. The image may be recorded during monitoring so that it can be referenced later. In addition, when a person moves around a place where a surveillance camera or the like is not installed or a place where the number of installations is small, it is possible to attach a small portable camera so that the person can receive this service.

  The 88th means is a monitoring service providing system configured by applying the 87th means, and the visual information processing configured by selectively applying the first to 60th means to a processing device of the system. An imaging apparatus configured by selectively applying the apparatus or the visual information processing system to which the apparatus or the 80th to 84th means are selectively applied.

  In the monitoring service providing system configured by applying the 87th means, the 88th means is configured by selectively applying the first to 60th means to a processing device of the system. Since a visual information processing apparatus, a visual information processing system to which the visual information processing apparatus is applied, or an imaging apparatus configured by selectively applying the 80th to 84th means is applied, the user's automatic operation is performed based on three-dimensional CAD data. Tracking becomes easy. In addition, since the surrounding situation can be seen with the three-dimensional CAD data, the situation on the site can be more accurately confirmed.

  Embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the basic configuration of first means and second means in the visual information processing apparatus of the present invention.

  The image information processing apparatus 100 generates CAD data by directly inputting image information obtained by the imaging apparatus 10 or indirectly through a recording medium, and stores the generated CAD data in the storage device 200.

  The imaging device 10 is a device such as a moving TV camera, and shoots every object to be converted into CAD data from various angles to generate image information.

  The image information processing apparatus 100 directly or indirectly inputs image information generated by the imaging device 10 and generates three-dimensional CDA data of any object included in the image information, and stores it in the storage device 200. Remember me. In addition, the image information processing apparatus 100 reads the ready-made CAD data stored in the storage device 200 as needed, compares the newly-generated new-CAD data with accuracy, and the accuracy of the new-generation CAD data is the ready-made CAD data. When it is higher than the data, the existing CAD data stored in the storage device 200 is updated with the new CAD data. Since the imaging apparatus 10 can freely move around in the environment, the imaging apparatus 10 can approach or leave an object in the environment. Since the object can be photographed with high accuracy when the imaging device 10 is close to the target object, for example, a portion that was visible as a single line at a distance is closer, or two appear as the viewing angle changes. Therefore, the image information processing apparatus 100 has a CAD data update function for updating the previously generated CAD data so as to be more detailed and so as to correct a part that has been erroneously recognized and generated earlier. Have.

  As a result, more accurate CAD data can be constructed as the imaging apparatus 10 is moved in various ways to capture an object in detail. In addition, since the imaging apparatus 10 can move to various positions, the image information can be generated by photographing the back surface, the bottom surface, and the top surface of the object, so that accurate as-built CAD data can be easily generated. Will be able to.

  FIG. 2 is a block diagram showing the basic configuration of the third means and the fourth means in the visual information processing apparatus of the present invention.

  The image information processing apparatus 100 inputs two pieces of image information obtained by photographing with the first imaging device 10 and the second imaging device 20 directly or indirectly through a recording medium. The first imaging device 10 and the second imaging device 20 are fixed at a predetermined interval so that stereo measurement can be performed. With this configuration, the image information processing apparatus 100 can generate CAD data while measuring distance information to an object by stereo measurement using two pieces of image information input from the two imaging apparatuses 10 and 20. It becomes possible. The other configuration is the same as the configuration of the visual information processing apparatus shown in FIG.

  Next, the internal configuration of the image information processing apparatus 100 will be described. The image information processing apparatus 100 basically includes an arithmetic processing device and an information (arithmetic) processing program. Here, the first means and the second means are configured by using a monocular imaging device 10 as shown in FIG. 1, and the third means and the fourth means are shown in FIG. In this way, the configuration is made using the compound eye (stereo) imaging devices 10 and 20, but the basic information processing flow of the image information processing device 100 is the same. After the basic process flow is described, the process in the case of a compound eye (stereo) will be described with reference to FIGS. Processing in the case of a single eye is processing that omits processing that can only be performed with a compound eye (stereo).

  In the basic processing flow in the case of a monocular, when processing is started, image information obtained by the imaging device 10 is taken in, processing such as differentiation processing and noise removal is performed, and a line segment is extracted. By tracking the start point and end point of a minute by tracking, etc., it is possible to obtain clear image information of corresponding points like the start point and end point between multiple pieces of image information taken from different viewpoints, so photo measurement The three-dimensional coordinates of those points are calculated by the method. Then, a line segment connecting the three-dimensional start point and the end point is defined as three-dimensional CAD data, and by defining a plurality of line segments in the same manner, line segments of various parts of the object in the space are defined. CAD data is generated. Here, if the imaging device 10 is further moved closer to the target object, the imaging device 10 can capture a larger image of the target object, so that the resolution, that is, the accuracy is improved. Then, the CAD data generation process is repeated, and if the line that was previously seen as one appears as two lines (CAD data update target discrimination function and CAD data accuracy processing function), the ready-made CAD data The one line segment in is updated to two line segments (CAD data update processing function).

  With such a function, the more accurately the target object is observed (photographed) by moving the imaging apparatus 10, the more accurate CAD data can be replaced.

  By doing this, one end point of the line segment disappears in the middle of tracking, and even if a corresponding point cannot be obtained in a predetermined number of pieces of image information, that point is still If the process of processing only points where a sufficient number of corresponding points are collected without processing and generating three-dimensional CAD data at those points is repeated, the imaging apparatus 10 continues to move. Therefore, by combining with the update processing function of CAD data, the number of points that are gradually measured increases and the CAD data also increases, and the CAD data becomes more accurate while being updated in more detail.

  Here, once the two lines appear as CAD data for two lines, and the imaging apparatus 10 takes a picture from a distance away from the object, the line 1 is again displayed. You may do the update process to return to the line segment of the book, but once it looks like two, even if it becomes one after that, two appear as one Therefore, a processing method that does not update the CAD data to one is appropriate for the purpose of more accurately creating the CAD data. In this case, for example, when calculating the three-dimensional coordinates, information on the accuracy of the calculation result at that time is stored together, and again, the same point around the same place (calculated earlier) It is also possible to determine whether a sphere having a predetermined radius around a certain point is the same place as a point that falls within the sphere, and the assumed radius depends on the accuracy of the previously calculated point. If the accuracy is good, it may be set to small, and if the accuracy is poor, it may be set large.) If required, compare the accuracy obtained this time with the accuracy obtained last time, and leave the higher accuracy. It is better to use a processing method.

  The concept of updating CAD data will be described in detail separately with reference to the drawings. Here, by providing a processing function to update, it is possible to generate only the points that can be generated at that time without applying a means to completely solve the problem of reliably tracking the corresponding points, which has been considered difficult in the past. By generating only line segments that can be generated and replacing them when they are generated more accurately, that is, by updating, it is sufficient to obtain sufficient CAD data of the required accuracy for the imaging apparatus 10. By moving to, it becomes possible to obtain CAD data with an accuracy sufficient to meet the purpose of use.

  Next, processing in the case of compound eyes (stereo) will be described with reference to FIGS. In the case of compound eyes (stereo), an embodiment will be described in which three-dimensional measurement using stereo and the imaging devices 10 and 20 are moved and combined with photo measurement as in the case of a single eye. Means for measuring three-dimensional (3D) coordinates, means for tracking, what is tracked as a feature point, CAD data is generated only for three-dimensional data, or two-dimensional image information based on two-dimensional Various methods of generating CAD data, such as using CAD data in combination, can be considered. Therefore, a processing speed, a memory capacity can be reduced, or an appropriate method suitable for the purpose may be applied. As described above, there are various ways of applying the present invention, but here, it will be described as an embodiment.

  FIG. 3 is a flowchart showing a basic flow of processing in the image information processing apparatus 100 realizing the third means and the fourth means.

  The start 100a may be started when the power is turned on, or may be started with a start button or the like, but is the start of processing. When the processing starts from the start 100a, first, the processing flow of the image information obtained by the imaging devices 10 and 20 is performed. The captured image information may be stored as raw image information in the storage device 200, or may be stored in an image-dedicated memory in the image information processing apparatus 100, but may be used for the next processing. It is necessary.

  Next, processing (1) 100c such as differentiation processing and noise removal of the captured image information is performed.

  Here, a specific example of the processing (1) 100c will be described with reference to FIG. Since the image information is stereo, there are two images. By performing differential processing or the like on the left raw image information (raw image) 101a, line image information (line image) such as processed image information 101c is generated. This process does not have to be a differential process, and the same result can be obtained by performing a process of extracting a contour. The right raw image information (raw image) 101b is processed in the same manner to convert it into line image information (line image) such as processed image information 101d. Here, what is important is that the start point, end point, etc. of the line segment are followed by tracking, so that the line segment extracted when the lighting state changes slightly or the viewpoints of the imaging devices 10, 20 change slightly. In order to obtain a stable processing result in which the length of the signal does not change drastically, it is necessary to consider a parameter for performing differentiation processing, a parameter for noise removal, a method, and the like. In addition, even if the end point of a line segment is determined using a threshold that may or may not be in contact with other line segments, even if the line segment is extracted in a separated state due to lighting conditions, etc. It is desirable to make corrections such as However, a line segment that cannot be seen if it is in contact with another line segment is left as it is until a visible line segment is extracted. In either case, it is preferable that the images are extracted in the same manner even if the viewpoints of the imaging devices 10 and 20 change slightly.

  In the next process (2) 100d, the line segments are numbered, or in the case of stereo, the left and right line segments are associated.

  Here, details of the process (2) 100d will be described with reference to FIG. Name and manage each line segment of the extracted line image information (line image). In the line image information (line images) 102a and 102b, outlines of rectangular parallelepipeds are extracted. Numbering is performed so that each line segment constituting the rectangular parallelepiped can be managed by giving names L1 to L9. In the case of the start point, end point, line type, and curve of the numbered line segment information on the line image information 102a and 102b, the function of the curve is stored in the internal memory or storage device 200 as a two-dimensional image database. And register. Here, a simple image of the two-dimensional image database will be described below.

  These data are increased to scenes (2), (3), etc. when the imaging devices 10 and 20 move to change the shooting scene.

By installing the left and right imaging devices 10 and 20 as close to the left and right eyes of a person as possible, the left and right image information is a little shifted, but the image information is almost similar (left and right Since the correlation is very good), the line segments and points corresponding to the left and right image data are associated with each other. In the case where the target object is in front and the left and right image information are extremely different, only the points of the portion that can be determined to be sufficiently the same as the partial matching are associated. The point where the correspondence could not be taken can be obtained from photographic measurement with time-series image information when the imaging devices 10 and 20 are moved later, so that processing cannot be performed simply because 100% correspondence cannot be obtained here. is not. The obtained association relationship is also registered in the left and right two-dimensional screen databases. For example, the line names may be the same as in the above example. The line segment name is the record number of the memory. Alternatively, the left and right data may be stored in physically separate memories in the storage device 200 and registered. Here, the coordinates of the start point and end point may be integers because they are addresses on the image memory. When the line type is a straight line, the coefficient of the fitting function may be omitted. If the curve is a quadratic curve, when fitting with a function of Y = aX ² + bX + c, there are three coefficients a, b, and c. When the line type is a circle or ellipse, the necessary number of coefficients necessary for the function are registered. A point may be defined as a short line segment whose start point and end point coincide.

  In the next process (3) 100e, in the case of a compound eye capable of stereo processing, at this time, each line segment is converted into three-dimensional data to generate a three-dimensional shape model.

  Here, details of the process (3) 100e will be described with reference to FIG. Since the line segment data corresponding to the left and right are already obtained as L1 to L9 in the stage of the process (2), the coordinates of the start point and end point of each line segment in the three-dimensional space are obtained in the stereo measurement of the compound eye. It can be calculated by calculating the coordinates. In this process (3), a three-dimensional database is generated based on the result of calculating the three-dimensional coordinates.

  The concept of this three-dimensional database is that the rectangular parallelepiped line segments L1 to L9 can be converted into three-dimensional line segment data, and if each line segment is connected to form a closed loop, surfaces S1, S2, and S3 are defined. It is a form that can. In addition, when there is nothing, the first plane (which may be considered as an infinite plane at infinity) S0 can be defined.

  Specifically, the start point, end point, line type, and line function of the three-dimensional line segments L1 to L9 are registered from the left and right associated image information. Next, the constituent line segments, surface types, and surface functions of the three-dimensional surfaces S1 to S3 forming a closed loop with the three-dimensional line segments are registered. The background is registered as the surface S0 for the time being (at first, the background surface is not yet determined, so there is no data content).

  The following is a simple image of this three-dimensional database.

  Since the above surface data is only simple plane data, it is also possible to have only the names of line segments constituting the surface as data.

  In the next process (4) 100f, if necessary, invisible data is generated. This is effective when CAD data is generated by considering all data as basic figures such as a cylinder, a sphere, a cone, and a rectangular parallelepiped. However, it is not necessary to do so.

  Details of the process (4) 100f will be described with reference to FIG.

  A group of faces that can be estimated to constitute a part of the basic figure is searched for from a group of three-dimensional planes generated based on the image information obtained by the first shooting. If not found, this process is passed. When it is estimated that the planes S1, S2, and S3 constitute a part of the basic figure, it is determined which basic figure is closest to the basic figure based on the coordinate data. Then, if it is determined that the planes S1, S2, and S3 are part of a rectangular parallelepiped, planes that are not visible are generated as estimated planes SS4, SS5, and SS6. At this time, line segment data of the invisible part is also generated as estimated line segments SL10, SL11, SL12. Furthermore, the configuration surface defines estimated solid data SV1, which is S1, S2, S3, SS4, SS5, SS6. These data are registered in the three-dimensional database.

  In this way, when stereo processing is used together, the CAD data before the imaging apparatuses 10 and 20 start moving, although there is an estimation part in part, immediately moves the arm toward the target object, for example, in manipulator handling etc. Can be used as control information (object search information).

  Here, the estimated data is managed so as to be understood as estimated data. For example, CAD data of the back side portion is generated based on image information obtained by shooting the back side portion of the object. In such a case, the estimated CAD data is replaced with non-estimated CAD data.

  As described above, the two-dimensional screen database is configured so that the necessary portions are processed in the processing (1) 100c, processing (2) 100d, processing (3) 100e, and processing (4) 100f in the flowchart shown in FIG. When the imaging devices 10 and 20 start to move by repeatedly registering in the three-dimensional database and taking in the image information from the imaging devices 10 and 20, a large amount of two-dimensional image data (scene) is accumulated. . Then, in the process 100g, the accumulation degree of the two-dimensional screen data is monitored, and the coordinates may be obtained by the photo measurement of process (5) 100h when it is accumulated to some extent.

  Details of the process (5) 100h will be described with reference to FIG. Although it may be considered as raw image information of either of the left and right imaging devices 10 and 20, when the viewpoint of the object of the raw image information 105a changes, the appearance changes as the raw image information 105c and the raw image information 105e.

  Here, the right side is a concept of line image information based on data registered in the three-dimensional database, and cannot be seen depending on the viewpoint angle of the imaging device 10 (20) like the line image information 105b, 105d, and 105f. The estimated line segments SL10 and SL12 generated by estimation from the above are not estimated at the stage of the image information 105d and are at the estimation level, but at the stage of the image information 105f, they are visible in the actually captured image information. Therefore, since these line segments can be generated directly based on the image information, the estimated line segment SL10 can be determined as the line segment L10, and the estimated line segment SL12 can be determined as the line segment L12. it can. In addition, as the two-dimensional image data based on image information taken from different viewpoints increases, photo measurement can be applied, so the spatial coordinates of the start point and end point defining each line segment are X, Y, and Z. Is updated so as to improve the accuracy of the CAD data by rewriting the coordinate data in the portion where the coordinates obtained by the photo measurement are more accurately obtained from the coordinate values obtained by the stereo measurement. At this time, if each coordinate value of the three-dimensional database is obtained by stereo measurement, the measurement accuracy at that time is stored in the database, and if the same point is newly recalculated in stereo measurement or photo measurement Then, the accuracy of the three-dimensional coordinate value calculated under the condition at that time is obtained by the method, and the coordinate value and its accuracy data are updated when the accuracy is better than the accuracy when the previous coordinate value is calculated. Incidentally, the accuracy of the three-dimensional coordinate calculation may be better for photo measurement than stereo measurement for a distant object. As a result, the CAD data can be automatically updated when the accuracy is improved.

The processing contents when processing (1) 100c to processing (4) 100f in FIG. 3 are repeated will be supplementarily described below.
(1) The imaging devices 10 and 20 that are initially stopped also move. Changes in the position and orientation of the imaging devices 10 and 20 may be calculated in this.
(2) Since two-dimensional screen data is added sequentially, data with little change in the middle is deleted so that the data amount does not explode (memory is insufficient).
(3) When a new line segment or point is generated in the same screen, it is newly registered in the two-dimensional drawing database.
(4) Even if the same line segment or point in the same screen is moved, it is tracked and managed as the same line segment or point. Correspondence as moving image tracking is managed in this two-dimensional screen database.
(5) At the same time, it is sequentially checked whether there is an error in the corresponding points of the left and right images, and if an error is found, the part related to it is immediately recalculated, and the two-dimensional screen database and 3 Modify the dimensional database.
(6) When a new line segment or point is newly generated, a new model is additionally generated in the three-dimensional database in processing (3) 100e and processing (4) 100f.
(7) If the two-dimensional screen data has been accumulated, the process (5) 100h is also performed to newly generate a new model in the three-dimensional database.

  As described above, when the imaging scene is image-processed and the three-dimensional coordinates are calculated, the lenses of the imaging devices 10 and 20 practically follow the tracking by shooting a wide range using a wide-angle lens. It is desirable to be able to generate and search CAD data for a wide range of environments in a short time while reducing the chance of losing sight of feature points. Distortion may occur, and accurate 3D calculation may not be performed. Therefore, when using a wide-angle lens, consideration must be given to using a wide-angle lens with less distortion, and consideration must be given to performing 3D calculation while correcting lens distortion by image processing.

  The distortion of the image information is caused by the refraction of the lens. Since the distortion characteristic of the image information is known as the lens characteristic from the relationship of the refractive index of the lens, correction according to the distortion characteristic of the lens is performed on the image. By performing it at the pre-processing stage of processing, 3D calculation based on image information without distortion after correction becomes possible. In order to perform such processing at high speed, it is preferable that the processing is performed by hardware such as an LSI circuit. Also, when using a zoom lens, the lens characteristics (correction parameters) vary depending on the zoom state, so it is desirable to use a correction circuit that takes in the zoom state and performs correction processing.

  FIG. 9 shows a concept of utilizing floating line segments generated as CAD data. In FIG. 9, for example, line segments L11, L12, and L13 are lines attached to each surface of the rectangular parallelepiped, and the coordinate data of the line segment data are coordinates included in a part of the plane of the rectangular parallelepiped. The shape of a rectangular parallelepiped is shown. Further, when a line is actually drawn, it can be used as line segment data indicating the feature of the object as pattern data.

  Therefore, floating line segments that do not constitute a plane generated when contour lines are extracted are registered and managed in the two-dimensional screen database and the three-dimensional database. Each line segment is registered and managed on which side. These floating line segments can express the shape of the surface by referring to the coordinates of the line segments in the surface when creating the surface data, and search for patterns and scratches as reference data. In this case, the data is necessary for matching. If surface data is attached to some surface, the relationship is registered and managed.

  FIG. 10 shows a concept when applied to a cylinder or a sphere. The cylinder has a line segment L7, and the sphere has a line segment L8. These are data indicating a part of the surface shape of a cylinder or a sphere as actual line segment data.

  In FIG. 10, line segments L1, L4, L5, and L7 are registered in the database as curves. The surfaces S2 and S3 are actually curved surfaces, but are unknown at this stage (may be planes). Since the positions of the lines L2, L3, and L5 change when the viewpoint changes in the surfaces S2 and S3, it can be seen that the line segments L2, L3, and L5 are the boundary lines of the curved surface. Estimate and define as a sphere, or estimate and define as a curved surface function from the coordinates of the floating line segments L7 and L8.

  FIG. 11 shows an example of the definition of the special curved surface as CAD data. For example, the leaf of a tree 900 and the like cannot be seen for each piece of image information taken from a distance. In this case, for example, it is assumed that the line segment L2, L3, L4, L5, L6 is attached to the special surface S1 formed by the contour line L1. Since each line segment has a three-dimensional coordinate value, the shape of the special curved surface S1 attached to the surface of the line segment can be considered to have a surface in the portion to which the line segment data is attached. By using such a concept of defining a special surface, it is possible to define a complex curved surface of a cloth or a kimono with a wrinkle or a surface such as a sand pile.

  Specifically, in the leafy tree 900, the contour lines L1 and L7 can be clearly extracted, but since there are no vertices or the like, it is difficult to define the shape, so the contour lines L1 and L7 are formed. The curved surfaces S1 and S2 are defined, and floating line segments L2, L3, L4, L5, L6, L8, L9, L10, and L11 are defined on the curved surfaces S1 and S2. The leaves sway in the wind, and the position and shape of the floating line segment change or disappear depending on the light condition, but it is sufficient if the data at that moment is considered positive. The reason that such a rough expression is acceptable is that data of a distant object is not directly used for control that requires accuracy. In handling control such as grasping a leaf with a manipulator, if the hand approaches the leaf, the imaging devices 10 and 20 also approach the leaf and obtain detailed image information for each leaf. This is because the required accuracy can be obtained with corresponding details.

  Similarly, the solid data can be defined as an object made of a curved surface on which floating line segments L2, L3, L4, L5, L6, L8, L9, L10, and L11 exist. This solid data can be used when examining a tree of such a shape of this size. If possible, it may be easier to mechanize the process if the curved surface is expressed by a function of the curved surface including those points than the coordinate data of the floating line segment.

  Such special curved surfaces are objects such as cloths, kimonos, bags, towels, etc., complicated curved surfaces, human fingers, arms, clothes, sandstones, distant mountains, curtains, etc. This is also the same expression.

  FIG. 12 shows an embodiment in which CAD data is updated in the first means, the second means, and the third means of the present invention.

  (A) is raw image information when an object is photographed by the imaging devices 10 and 20 located far away from the object. When the object is photographed close to the object, (B) is obtained. This object is like a desk, for example.

  When CAD data is generated based on raw image information (A) obtained by photographing this object from a distance, a rectangular parallelepiped composed of line segments L1 to L9 is generated as shown in (a). Therefore, when CAD data is generated based on raw image information (B) obtained by shooting the imaging devices 10 and 20 close to an object, data as shown in (b) is generated. In this case, new line segments L10, L11, L12, L13, and L14 are generated and added, and the coordinate data of the start and end points of the line segments L1 to L9 are also measured nearby, so that the accuracy is higher. Updated to coordinate data.

  Specifically, one outline generated based on raw image information obtained by photographing an object from a distance improves the resolution according to the raw image information obtained by approaching and photographing. There may be two contour lines, which may result in more accurate shape data.

  Here, the description will be made by updating the two-dimensional screen database. Similarly, all the related three-dimensional databases are also updated.

  The line segments L1, L2, L3, and L5 generated based on the raw image information obtained by photographing the object at a position far away from the object are each one, but obtained by photographing at a close position. There are two line segments generated based on the raw image information. The line segments L1, L2, L3, and L5 are not deleted but updated to the newly obtained coordinate values as more accurate data of the nearest line segment. If there is no close line segment, delete it and create a new one. New line segments L10, L11, L12, L13, and L14 are newly numbered and additionally registered.

  In this way, if not only the generation of CAD data but also the update is automatically performed, the generation of CAD data based on the raw image information obtained by photographing by the movable imaging devices 10 and 20 is performed. , 20 moves, the better the accuracy. Also, since CAD data can be generated with a certain degree of accuracy even at the initial stage, this CAD data can be used from an early stage for robot control and the like.

  FIG. 13 is a conceptual diagram showing an embodiment of the fifth means. Here, it is assumed that the imaging devices 10 and 20 generate CAD data based on image information obtained by shooting while moving around a cylindrical object (cylinder) 901 as a target. Further, the cylinder 901 is an object on which a character 101A “A” is drawn.

  In such a state, in the imaging devices 10 and 20, the side line of the cylinder 901 and the upper or lower disc ridge appear to be an ellipse. However, all of these lines are outlines, and actual lines do not exist there. Therefore, when generating the three-dimensional CAD data, if the image is generated by densely capturing the image information of the ridges of the cylinder 901 and detecting a lot of contour lines around the periphery, the data is generated in the three-dimensional database 200G. In this case, CAD data of a wire frame like a cylinder 902 is generated.

  In addition, the line segment constituting the letter “A” drawn on the surface of the cylinder 901 is an actual line that is a line segment actually present there. For example, it is generated as line segment data constituting the character “A” 902A in the CAD data. If such a contour line and an actual line are properly used when using CAD data, the CAD data can be used efficiently. For example, since the line segment data of the contour line does not actually exist, it becomes an obstacle when searching for the line segment data of the character “A”. In addition, if you want to refer to the shape data of a cylinder, you can get more detailed data by using many contour lines, so it was determined that this is an actual line when generating the data. CAD data is generated so as to be defined as an actual line at a stage. In this case, when the imaging devices 10 and 20 are moved, the outline of the side surface of the cylinder 901 does not change even if the imaging devices 10 and 20 are moved, whereas the letter “A” is When the imaging devices 10 and 20 are moved, the visible place changes. When the positions of the line segments before and after the movement of the imaging devices 10 and 20 are compared after being converted into three-dimensional data, the outline line segment before the movement is not visible as a line segment at the position after the movement. However, the character A, which is an actual line, identifies that the line segment constituting the character “A” is an actual line segment from the characteristics that exist in the same place even when the imaging devices 10 and 20 move. be able to. Further, in the case of a side contour line such as a cylinder 901, the position of the contour viewed by the compound-eye imaging devices 10 and 20 is slightly shifted. However, when generating the contour line by stereo measurement, Since it affects as an error, it is easy to be careful when using it if it is distinguished that the contour line includes such an error.

  An example of the concept of a three-dimensional database is shown below. One method for distinguishing among line types is shown below. A line segment L1 indicates an actual line, and a line segment L2 indicates an outline.

  Next, an embodiment of the sixth means will be described. The method for generating the basic figure of the rectangular parallelepiped as the solid data SV1 has been described with reference to FIG. 7, but in a system in which the imaging device moves and updates the CAD data, the imaging device sees an object from a distance. Even if it is a rectangular parallelepiped, when viewed close to it, for example, when the corner of the rectangular parallelepiped is round like a dice, it is not appropriate to define it as a rectangular parallelepiped. Also, when you look at a large building from a distance, it looks like a rectangular parallelepiped, but if you get close, there is a door, and when you go inside, the building that was initially visible as a rectangular parallelepiped becomes an object surrounded by a plane like a wall. Furthermore, when viewed closer to the wall, it becomes a flat rectangular parallelepiped having a plurality of thicknesses. In such a system, it is not always necessary to drop it into a solid model. Therefore, considering the minimum necessary data configuration, it is possible to raise points, lines, and planes, so it is advantageous from the viewpoint of processing speed to create a system that generates and updates the minimum necessary plane data. is there.

Next, an embodiment of the seventh means will be described. The database already described has a form that expresses line data as a function and a form that also expresses surface data as a function. However, expressing surface data as a function saves in terms of processing complexity and data memory saving efficiency. Is less effective. Therefore, it is proposed that the data amount be greatly reduced by expressing the function of at least line data. For example, when processing is performed for a normal environment, an enormous number of line segments are extracted. If you try to express the shape of these line segments as a straight line, for example, a line segment with a circle or a complicated bent shape will define many points on the line segment, and a straight line between those points will be defined. It must be connected with line segments to express a circle or a bent shape, and a lot of point data is also required, so a huge memory capacity is required. Therefore, the amount of data can be greatly reduced by defining the curve as a function and expressing the curve with appropriate coefficients (a1, b1,... In the above data example).
An embodiment of the eighth means will be described. In this embodiment, color information is given to at least surface data in CAD data to be generated and updated. The concept is as follows.

  Of course, it may be possible to distinguish subtle intermediate colors by color code instead of red, blue, etc. Thus, if there is color information in the CAD data, it is only necessary to search for the CAD data having the red color information when searching for the red one, so that the efficiency in handling the CAD data is improved. In addition, if the point where the color changes in the color information is positively defined as the line segment data of the boundary line, the line segment data showing more shapes can be obtained, and the color distribution state can be defined in more detail. It becomes possible.

  FIG. 14 illustrates the concept of extracting a contour using color information. Reference numeral 110a is raw image information. A small frame of each mesh is a portion corresponding to one pixel. It shows the situation that there are various color surfaces and line segments as image information.

  Considering the creation of a two-dimensional image database on this image data, for example, for each pixel, the entire screen is processed while comparing the color of the surrounding pixels with its own pixel, and the pixel portion where the color changes If the boundary line is defined in the table, the result shown by reference numeral 110b is obtained. If it is further defined symbolically, the surface data 110c, 110f, and 110i define colors in their respective closed loops, and color information is also defined in the surface data. In addition, in the boundary portion, only the contour line data 110e or the line data 110d is defined in the surface 110f when only the line is used, and the line data 110h is also independently defined. The contour line 110e here indicates a boundary line of different colors, so if it is a painted pattern, it may be defined as an actual line. Here, a virtual point 110j is defined in the heel portion of the screen so that the plane data 110i can be defined. The concept of these diagrams is a concept of a two-dimensional image database. A two-dimensional image database may take a method of defining surface data in addition to line segment data, and surface data is defined only in a three-dimensional database. You may do it.

  Here, a supplementary description will be given of the procedure of the processing in FIG.

  The color and brightness of adjacent pixels are compared, and if it changes greatly, it is determined as a contour. Similarly, when searching around, a closed loop contour is created. It is registered as contour line data (colorless at the border) and closed loop surface data. Those without a region (area) are registered as line data.

  Next, an embodiment of the ninth means will be described. An example of CAD data is shown below.

  Here, the maximum brightness is set to 100, the background surface S0 is illustrated as 50 brightness, the red surface S1 is illustrated as 20 brightness, and the blue surface S2 is illustrated as 100 brightness. By defining in this way, for example, when the surface S2 is a surface that shines with a blue lamp, a method of using the brightness data 100 to determine that the blue lamp is lit is used. Will be able to. Here, when CAD data is generated, if brightness information of the surface of the image information obtained by the imaging devices 10 and 20 is referred to and the information is registered in the CAD data, brightness information is added to the CAD data. Is possible.

  Next, embodiments of the tenth means to the twelfth means will be described. An example of CAD data is shown below.

  In the above example, the plane data S1 and S2 indicate portions where two types of data, that is, data generated at time 13:15 and data generated at time 13:20 exist. In this way, by registering the time data at which the data was generated, for example, the surface data S2 can easily indicate that the blue lamp has disappeared because the brightness has changed from 100 to 20. To be able to recognize. If the data different from the CAD data generated at the previous time is left, it is possible to recognize the change state and history by examining the same CAD data. In addition, since the memory capacity becomes insufficient when data is added in this way, for example, duplicate data is automatically deleted after a predetermined time of 1 hour or 10 minutes from the generation time. By programming in such a way, it is possible to prevent the memory capacity from exploding. For example, automatic erasure processing can be performed by constantly checking the time of CAD data and erasing data after a predetermined time has elapsed.

  With reference to FIGS. 15 and 16, embodiments of the thirteenth to eighteenth means will be described.

  FIG. 15 shows image information obtained by photographing the scene in which the virtual image 903b of the person 903a and the virtual image 903d of the sun 903c are reflected in the car 903 and the shadow 903e of the person 903a is reflected on the ground. The example which produces | generates CAD data based on this is shown.

  Here, the generated CAD data is defined such that the surface S1 forming the curved surface of the body of the car 903 is defined, and the virtual image 903b, 903d, the mark 903f of the car 913, etc. are attached to the surface S1. However, since the mark 903f is a mark that is actually attached to the surface S1, and the virtual images 903b and 903d are data that is not actually attached to the surface S1, the CAD data is used. It is convenient if it is easy to distinguish.

  Therefore, as processing, if the person 903a moves, the virtual image 903b moves, or if the imaging devices 10 and 20 move, the virtual image 903b on the surface S1 is also motivated, while the mark 903f actually attached to the surface S1 is Since the position in the surface S1 does not change, paying attention to the CAD data attached to the surface S1, if there is CAD data that moves together in these, manage them as a group To do. That is, since the CAD data of the line segment and the surface constituting the virtual image 903b move together when the person 903a moves, such data is detected and managed together (grouped). Similarly, the virtual image 903d moves in the plane S1 if any of the sun 903c, the vehicle 903, and the imaging devices 10 and 20 moves, so that the change in the position can be detected with reference to the coordinate data. Then, it is possible to detect and group similarly. Further, when the sun 903c or the person 903a moves, the shadow 903e attached to the ground surface S0 also moves, so that the line segment data and surface data constituting the shadow 903e can be extracted at that time, and the ground S0 is actually It can be distinguished from other attached CAD data. Here, the virtual images 903b and 903d move when the imaging devices 10 and 20 move, but the shadow 903e does not move. Therefore, the captured virtual image and shadow may be managed separately. Further, for example, the dust attached to the window glass of the car 913 is the CAD data actually attached to the surface of the window glass, but the CAD data visible behind the window is temporarily attached. Since only the object can be seen and the object in the back is actually seen through, this also moves when the imaging devices 10 and 20 are moved. Therefore, it may be temporarily distinguished as a virtual image.

  Next, a description will be given of an embodiment in which a virtual image of an object that is visible even in a virtual image and a virtual image of an object that is visible are distinguished and managed.

  FIG. 16 is a diagram for explaining an embodiment in the case of distinguishing and managing a visible virtual image and a transparent virtual image. The scene shown in FIG. 16 is an example of a room in the house, with a window 910a in the back, and a picture 910b and a mirror 910c. Consider generating CAD data based on image information obtained by photographing the scene of this room while moving an imaging device (not shown).

  In the generation of CAD data, if the line segment becomes CAD data and the portion constituting the closed loop is generated as surface data, in this scene, the surface S1 formed by the inner frame of the mirror 910c, the window 910a CAD data such as the surface S2 formed by the frame and the surface S3 formed by the inner frame of the frame of the picture 910b is generated. Then, CAD data of the cylinder V1 placed on the floor is also generated, but the cylinder V2 reflected on the mirror surface S1 is also generated as CAD data.

  Since the cylinder V2 reflected on the mirror surface S1 moves separately from the dust attached to the surface of the mirror surface S1, it can be distinguished as a virtual image. Can not do it. The same applies to the virtual image of the box V3 that appears to be attached to the surface S2 of the window 910a. As a process for determining this, for example, assuming that the surface is a mirror surface, from the positional relationship between the imaging device and the surface, if the surface is a mirror surface, there is an entity at the position to be reflected and captured. Therefore, it can be determined by checking the existence of this entity. In this embodiment, since the cylinder V1 which is the substance of the cylinder V2 attached to the surface S1 is at a predetermined position, it can be determined that the surface S1 is a mirror surface. Similarly, since the virtual image that appears to be attached to the surface S2 of the window has no entity when it is assumed to be a mirror surface, it can be identified as an object that can be seen through.

  In addition, a triangular figure S4 is attached to the surface S3 of the picture 910b, and this moves together with other pictures or dust attached to the surface S3, so that it can be identified as not a virtual image.

  In this way, it is determined whether the surface is a mirror surface or a transparent surface by comparing it with the surrounding entities, and if it is determined to be a mirror surface, the surface data is added to the database as a mirror surface and registered in the database, Register and manage as a group. In this way, once it is discriminated as a mirror surface, the same discrimination process does not have to be repeated each time a different virtual image appears, so that it can be processed efficiently. If the line segment and surface data of the virtual image appearing on the mirror surface are created as environment data as they are, there is a contradiction with the actual data, so it is better to manage them separately or delete them.

  Similarly, if a transparent surface is registered separately as a transparent surface, the subsequent processing can be processed as a transparent surface from the beginning.

  What is important here is that if a decision flag is attached at the stage where it is definitely decided, a safe system can be constructed even for important matters.

  In addition, even on the same surface, there are cases where there are a maximum of three groups, one that is actually attached such as dust, one that is reflected as a mirror surface, and one that is seen through. Considering such a case, if the identification data of the surface is also provided with the identification of the half mirror, the management thereof becomes easy.

  If there is no data with dust or the like, there may be an opening with nothing, so it is preferable to provide an identification that there is a possibility. In practice, there are surfaces with different degrees of reflection and transmission. For example, the reflectance and transmittance are obtained from the ratio of the luminance of the real image and the virtual image, and the numerical values are added to the surface data. It is good to do so. Moreover, it is good also as a system which can input and set an exact transmittance | permeability and reflectance later manually. Further, when the transmittance and reflectance differ depending on colors, the characteristic information may be provided for each color (for each wavelength). In addition, when painting a car body or the like, there is a surface that changes its color depending on the viewing angle and the angle of light. In such a case, for example, when it is viewed from this position at such an angle, what color it looks like is registered together in the color information of the CAD data of the surface together with such a condition. Also good. If at least a flag indicating that the color changes depending on the viewing position is registered, it is possible to prevent erroneous recognition in the case of searching for any color car using CAD data. It will be possible to realize a system that can.

  FIG. 17 shows a flowchart of such mirror surface discrimination processing, and FIG. 18 shows contents that can be discriminated by this mirror surface discrimination processing.

  An embodiment of the nineteenth means to the twenty-first means will be described with reference to FIG.

  The scene of the image information (image) 111 a shows an example of image information obtained by photographing with the imaging devices 10 and 20. The image shows dust 911a and people 911b attached to the lens. Further, when an image pickup device such as a CCD of the image pickup devices 10 and 20 is damaged by radiation, an image similar to a state in which dust is attached to the lens is obtained. If CAD data is to be generated based on such image information, dust images are also processed. Therefore, if the number increases, wasteful processing increases.

  Such dust does not move in the image even if the imaging devices 10 and 20 move. In the image information 111a, if the imaging devices 10 and 20 are moved, the person in the scene moves, but the dust does not move.

  Therefore, by utilizing such characteristics, the image information that does not move even when the imaging devices 10 and 20 are moved is erased at the stage when the two-dimensional image information is taken in, and the object of the subsequent CAD data generation processing So that processing when there is a lot of such garbage can be performed efficiently.

  Further, in the scene of the image information (image) 111b, a noise image 911c generated in the electric circuit of the imaging devices 10 and 20 is shown together with the target person 911b. If the number of the noise images 911c increases, wasteful processing is performed in the CAD data generation processing. Therefore, the noise information 911c is removed immediately after the image information is captured and excluded from the CAD data generation processing. For this purpose, for example, in this case, since there are many noise images 911c that move in the same pattern regardless of whether the imaging devices 10 and 20 are moved or stopped, they may be removed using the features. . An image that is long and horizontally and periodically generated randomly is extracted, or a noise pattern that appears to move when the imaging devices 10 and 20 are not moved is extracted, and such a pattern is always removed. This can be realized.

  Further, the image information 111c is an example of a scene where there is a person in the rain, but in this case, a lot of rain 911d appears in noise. Like the noise, these may be a method of removing first by using a feature of long noise when two-dimensional image information is taken in, and CAD data of rain 911d can be easily generated. Since it can be generated, its characteristics, that is, thin and vertically long data is deleted or managed in another group, for example, when using CAD data of person 911b, CAD data only for person It is also possible to make it easy to handle.

  For example, when CAD data is generated based on image information obtained by mounting the imaging devices 10 and 20 on an underwater ROV and floating in water, such processing floats to some extent. When it is possible to efficiently handle only necessary CAD data by extracting and removing CAD data such as those that move together along the flow, such as trash Can be used.

  FIG. 20 is a block diagram showing an embodiment of the twenty-second means to the twenty-sixth means. The imaging device 10a, the image information processing device 120a, and the wireless device a are connected and installed in a place where the cylindrical object 901a exists, and the image information processing device 120a includes a storage device 200a that stores the generated CAD data. . Similarly, the imaging device 10b, the image information processing device 120b, and the wireless device b are connected and installed where the rectangular object 901b is, and the generated CAD data is stored in the image information processing device 120b. There is a storage device 200b. In another place, there is another image information processing device 120c connected to the wireless device c, and among them is a storage device 200c.

  In such a configuration, CAD data is generated based on image information obtained by the imaging devices 10a and 10b by the various means described above. Specifically, CAD data of the cylinder 902a is generated in the storage device 200a, and CAD data of the rectangular parallelepiped 902b is generated in the storage device 200b. Then, the CAD data 902a (902b) generated by one image information processing apparatus 120a (120b) can be converted into another image information processing apparatus by allowing the CAD data to be transmitted and received through the wireless apparatuses 121a to 121c. It becomes possible to generate CAD data integrated by using in 120b (120a). If the integration of CAD data is performed by the visual information processing device 120c located at another place, the image information processing devices 120a and 120b can omit the CAD data integration processing and efficiently perform the CAD data generation processing. You can do well. In this integration processing, the CAD data 902a generated by the image information processing apparatus 120a is transmitted to the image information processing apparatus 120c, and the CAD data 902b generated by the image information processing apparatus 120b is also transmitted to the image information processing apparatus 120c. The processing device 120c executes CAD data integration processing. The integrated CAD data is generated in the storage device 200c. In this case, it is necessary that the coordinate systems generated by the image information processing apparatuses 120a and 120b are the same or can be matched before integration. For example, if the current position and orientation of the imaging devices 10a and 10b are input in the common absolute coordinate system in the image information processing devices 120a and 120b when the power is first turned on, both pieces of image information are stored. The coordinate systems of the CAD data generated by the processing devices 120a and 120b can be matched. Further, it is also possible to generate CAD data by wrapping the target object at the same place in both the imaging devices 10a and 10b, and calibrate the coordinate system so that the CAD data generated by wrapping matches.

  If the image information processing apparatus 120a and the image information processing apparatus 120b directly transmit and receive CAD data, the CAD data can be directly shared. Therefore, another image information processing apparatus 120c that performs integration processing. Can be made unnecessary. In such a case, it is necessary to provide a function for integrating CAD data in the image information processing apparatuses 120a and 120b.

  Further, the image information processing apparatuses 120a and 120b generate each other by returning the result of integration in the image information processing apparatus 120c that executes another integration process to the image information processing apparatuses 120a and 120b. If CAD data can be used together, the result of the integration processing can be used even in the image information processing apparatuses 120a and 120b having no integration processing function.

  Data transmission / reception between the image information processing apparatuses 120a, 120b, and 120c may be performed via a wired cable. However, by using the wireless apparatuses 121a to 121c, shooting is performed while moving the imaging apparatus 10. The mobility in generating CAD data based on the obtained image information is improved, and the target object can be easily photographed from various angles. In addition, even when CAD data is generated and integrated over a wide range, it is possible to generate and synthesize a wide range of CAD data without worrying about the cable length.

  21, FIG. 22, FIG. 23, and FIG. 24 are block diagrams showing embodiments of the 27th to 29th means.

  FIG. 21 shows a case where another CPU 300 is connected to the image information processing apparatus 100, and the position and orientation information of the imaging devices 10, 20, that is, the camera calculated in the image information processing apparatus 100 each time are stored in another CPU 300. This is a device that can be taken out with Here, for example, by outputting the camera position and orientation information from the image information processing apparatus 100 at regular intervals, the CPU 300 can obtain the camera position and orientation information at that interval. With this configuration, the position and orientation information of the camera can be used in an application system configured using the CPU 300. For example, in the case of a robot arm control system, if the CPU 300 controls the robot arm, the CPU 300 can be attached to the tip of the robot arm so that the CPU 300 can detect the position of the tip of the robot arm. And attitude information can be obtained, and the result can be applied to control.

  FIG. 22 shows a configuration in which the CPU 300 of the application system inputs a command signal from the CPU 300 to the image information processing apparatus 100 in order to obtain the camera position and orientation information, and the camera position and orientation information is received as image information in response. The apparatus is configured to be acquired from the processing apparatus 100. Therefore, the image information processing apparatus 100 can output the camera position and orientation information to the CPU 300 only when the CPU 300 requires it.

  FIG. 23 is configured so that a CAD data search command signal can also be input from the CPU 300 of the application system to the image information processing apparatus 100, and as a response, information (shape or shape of CAD data searched by the image information processing apparatus 100 may be used. , Position and orientation information) may be acquired by the CPU 300. As a result, the CPU 300 side can easily obtain both the position and orientation information of the object to be controlled and the position and orientation information of the other party that is necessary for the control, so that information is used for the control. Will be able to.

  In FIG. 24, not only the search command signal but also a command signal for outputting camera position and orientation information is output from the CPU 300, and a search result and camera position and orientation information are acquired from the image information processing apparatus 100. It is the apparatus comprised in. In the apparatus configured as described above, for example, when a cup is held by an arm, a camera is provided at the tip of the arm, and the image information processing apparatus 100 is made to search for CAD data of the cup to be held by the arm. The position and orientation information of the cup can be obtained from the CAD data search result, and the position and orientation information of the tip of the robot hand can be obtained as camera position and orientation information and used for control. . Since the camera position and orientation information is information calculated when CAD data is generated, there is an advantage that the camera position and orientation information can be obtained in a short time because it does not require another time for calculation processing. Of course, the position and orientation of the target cup can be searched using CAD data, and the robot hand can also be searched from the shape data of the hand to obtain the position and orientation information of the hand.

  FIG. 25 is a flowchart showing an embodiment of processing of the image information processing apparatus 100 when the twenty-seventh to twenty-ninth means are implemented as a visual information processing apparatus dedicated to robot control.

  The processing of the image information processing apparatus 100 starts from start 123a and performs command input processing 123b from the CPU 300 or the like in the application system. Here, in the case of a command called CAD data search, information on what kind of CAD data is input.

  Next, the processing proceeds to image information processing, and first, image information capturing processing 123c is performed. Subsequently, three-dimensional measurement, 3D-CAD data generation processing, camera position and orientation calculation processing 123d are performed. Then, a CAD data search process 123e is performed.

  And the process 123f which outputs the position and attitude | position information of CAD data searched with the camera to CPU300 is performed.

  Thereafter, the process repeats from the command input process 123b. By making this repetitive cycle be performed at a high speed of several tens of milliseconds, the CPU 300 that controls the robot or the like in real time can easily acquire information used for controlling the robot.

  Specifically, as shown in FIG. 25, the camera 123g provided in the robot hand unit and one of the objects in the environment, for example, so that the position and orientation information of the cylinder 123h can be known in real time. Therefore, it is possible to easily realize control for guiding the robot hand to hold the cylinder 123h.

  With reference to FIGS. 26 and 27, embodiments of the 30th means, the 33rd means, the 34th means and the 39th means will be described.

  FIG. 26 shows a scene where a car is running on a road in a city with traffic lights and buildings, and an example in the case of searching for a signboard of the HITACHI building 904 in the scene will be described.

  As a method of searching for CAD data, a method of searching for a thing close to the basic figure based on the basic figure is adopted. In this case, if the size of the signboard is known, a rectangular body having a predetermined dimension is prepared as the reference CAD data 904a as a basic figure, and the CAD data that is close to it is searched from the generated CAD data. By doing so, the signboard of the HITACHI building 904 can be easily searched.

  In the example shown in FIG. 27, since only a limited shape can be searched using only basic figures, a part of the generated CAD data is extracted in advance and input as reference CAD data 904b for search. To do. In this case, since the CAD data itself is generated, there is always a portion that matches with high accuracy, so that the search can be performed more reliably and accurately. In addition, it becomes possible to search a part of a block of CAD data having any shape not included in the basic figure.

  Here, as a method of reading the search data in advance, it is an example of the thirty-third means. For example, a cuboid that can be changed to an arbitrary size is displayed in the generated CAD data, and the cuboid of the cuboid is displayed. There is a method of reading out CAD data as CAD data for search in an area. According to this method, a necessary part can be easily taken out. If the area 904A is designated with a size that encloses the HITACHI building 904, the CAD data for search including the signboard portion can be read out. By searching with this data, the character data of the signboard is also used as characteristic information when searching, so that the target search target can be found more reliably. Of course, if various locations are read out, it is possible to construct a system in which the locations can be searched and confirmed immediately when necessary. The area may not be a rectangular parallelepiped but may be a sphere that can be set to an arbitrary size. Since it is not necessary to specify the posture if it is a sphere, the operation becomes easy.

  A thirty-fourth means is a method of first searching for a signboard with a basic figure and then reading a larger area including the signboard as search CAD data. With this method, it is possible to search using only the basic figure at first, and then to use the part of the signboard including characters of the signboard for searching. Of course, even if it is not a basic figure, if a similar place is searched and the vicinity of the similar place is read as search CAD data, the search CAD data suitable for searching for the place is used. Can be obtained.

  In this way, by using the thirty-fourth means, even if there is only a basic figure at first, it is possible to search for a similar place gradually and use it as new search CAD data. It is also possible to create a data collection.

  The thirty-ninth means is to search for a basic figure, for example, if the size of the rectangular parallelepiped of the signboard is unknown, specify the approximate ratio of the vertical and horizontal lengths indicating a long rectangular parallelepiped. This is a technique for searching for a plurality of rectangular parallelepiped objects close to the ratio. This method is effective when it is desired to search from a large size to a small size if the shapes are the same. This is useful when searching for valves with the same shape and various sizes, such as valves and pipes. In addition, even when searching from children to adults in a human-like form, by allowing a certain degree of matching, the same search data can be specified in a size-free manner to search for large and small similar items. Will be able to. Since the search data need not have the same number as the size, memory can be saved.

  With reference to FIG. 28, an embodiment of the 31st means and the 32nd means will be described. The imaging apparatus 10 enters the room 905a together with the object 905 to be imaged, images the object 905 from various angles, up and down, left and right, generates image information, and generates CAD data. In this case, the object 905 to be imaged is a flower.

  Since a flower in a general environment is shaded by various things, it is difficult to generate CAD data of the entire shape of the flower, but by using this method, accurate CAD data 905c of the object 905 to be photographed is used. Can be easily obtained. Further, if no other object is present in the dedicated room 905a, the other object will not be mixed even if the target object 905 is photographed from any angle. The interior of the room 905a may be white or blue on the floor, wall, and ceiling, or a grid such as a grid is drawn on the floor, wall, or ceiling of the room, and the grid is generated when CAD data is generated. May be used.

  With reference to FIGS. 29, 30, 31, and 32, embodiments of the 35th to 38th means will be described. 29 to 32 are for explaining an example of the reference CAD data search process 123e in the basic process described with reference to FIG.

  FIG. 29 is an arrangement of search example conditions. The image information (video) of the current imaging device (camera) includes four rectangular parallelepipeds, and three of the rectangular parallelepipeds are stacked in three stages. It is assumed that CAD data has been performed to some extent, and CAD data having a similar environment has already been generated. Specifically, it is assumed that solid data SV1, SV2, SV3, V4 corresponding to each rectangular parallelepiped has been generated, and data with solids SV1, V4 on the plane S0 has been generated. In this state, the reference CAD data to be searched is a two-layered rectangular block.

  An example of the process (1) in the search process 123e will be described with reference to FIG. This process is a process of calculating a feature amount of search CAD data (reference CAD data). There are various features that can be considered depending on CAD data. Here, volume, slenderness ratio, vertices, line segments, number of faces, number of irregularities, number of basic shapes (two cuboids), line segments, These include surface characteristics (functions for curved surfaces), color information, and the like. These calculation processes are performed only once when new search CAD data is input.

  An example of the process (2) in the search process 123e will be described with reference to FIG. Specifically, it is a process of searching for CAD data having the same feature quantity as the feature quantity extracted in the process (1) from CAD data in the field of view that is currently visible.

  In this example, since there are four cuboid data, each of the four cuboids (candidates 1 to 4) and the stacked cuboids have two combinations of two (candidate 5 and candidate 6) and three more stages. Since there are a total of seven candidates for the combination of the overlaps (candidate 7), if the feature amounts of the seven candidates are compared with the feature amounts of the reference CAD data to be searched, candidate 5 is the best match. In a simple environment in which a basic figure can be searched, the candidate CAD data with the most approximate basic shape is the search result, and as a result, the center position and orientation information of this candidate CAD data is output. You can do that. This may be performed all at once in the basic search process 123e in FIG. 25, or may be performed in several steps. That is, the search result may be output about once every 10 generation cycles of CAD data, or may be output every time.

  FIG. 32 is a processing method that is particularly effective when the environment is not only a basic figure. Reference CAD data and candidates are selected after selecting one candidate that is closest to the process (2) in the search process 123e, or in some cases, selecting a plurality of candidates. This is a process (3) of searching for a more appropriate candidate by performing pattern matching of CAD data.

  This example is a 3D potential matching method, in which reference CAD data is placed at the most probable position and orientation of the closest candidate to perform final matching. If the degree of matching is equal to or greater than a predetermined value, the object is specified as an object to be searched, and if it is less, the search is continued as no.

  In the matching processing, the search speed can be increased by extending the pattern potential matching method, which has already been established by the two-dimensional processing, to three dimensions, or using the eigenvalue space matching method. Here, the potential matching method is used to extend the two-dimensional potential matching processing described in Japanese Patent Laid-Open No. 02-137072 to three-dimensional processing. The eigenvalue space matching method is described in the literature of IEICE Transactions D-II Vol. J80-D-II No. 5 PP. The method described in 1136 to 1143 is applied. In any case, it is preferable to first use a variety of matching methods for final determination by narrowing down candidates by feature amount. The matching method used here is not limited to the above. A matching method that is performed in combination with a method capable of high-speed and high-precision matching may be used. Further, a method of using a combination of a plurality of matching methods may be used.

  Here, the search CAD data may be three-dimensional or two-dimensional. The two-dimensional CAD data is a pattern, a picture, a signboard character, or the like. That is, they are given as CAD data in which various surface data and line segment data are attached to one surface data. Color information may be attached to the surface. Even if such CAD data for search is given in two dimensions, the object to be searched is the space of the three-dimensional CAD data, so that the most matching portion of the two-dimensional plate is searched from the three-dimensional space. Become. In that case, the two-dimensional plate searches for a matching place by changing its orientation and posture in various directions. As in the case of three-dimensional search CAD data, it is preferable that the scale factor is variable so that the same plate, that is, a pattern or a mark, can be searched from large to small.

  Here, a supplementary explanation of the matching search will be given. In order to reduce the amount of data generated in the CAD data of the generated environment and the input CAD data for search, the line data is expressed by the start point, the end point, the line type, the function parameter of the line, etc. It is expressed by the name of the line segment to be configured, etc., but at the time of the search, as for the portion to be matched, if it is a line segment, it will be a plurality of point data on the line, and if it is a plane data, it will be Conversion to point data may be performed to perform matching processing between the point data, that is, processing for obtaining correlation. The point data to be replaced from the surface data having the color information is also provided with the color information, and matching is performed in consideration of the color information. When the matching process is finished, the replaced point data can be deleted. By replacing the data with point data each time, the amount of data as CAD data that should normally be kept can be reduced.

  Moreover, although it is the density at the time of replacing with a point, it is good to change appropriately according to matching accuracy. Basically, replacement is performed at equal pitches and equal intervals, but it is preferable to increase the density of locations of complex shapes and complex color patterns and to coarsen the locations of color data of simple shapes and simple patterns. In addition, if solid CAD data is used for management and input / output, the solid data is also replaced with point data in the solid space with a constant density or by providing dense and coarse portions. It is better to match the point data. By replacing with point data, a general-purpose matching process can be used in common. Here, the method of replacing with point data has been described, but it may be used as long as it can be matched on a function basis.

  An embodiment of the 40th means and the 41st means will be described with reference to FIG. When searching for reference CAD data from among a large amount of CAD data generated extensively, in order to increase the search speed, the entire space where the CAD data is generated is divided into block addresses, and the partition addresses are designated. The search range should be narrowed down.

  FIG. 33 shows the concept of searching by dividing a space into blocks of the same size. The camera which is the imaging device 10 exists among them, and the camera 10 has shown the condition which image | photographs the range of the visual field 10V.

  In the case of searching, there are a method of searching a range that is visible with the camera 10 and a method of searching a range that is not visible. Of course, there is a method of searching while CAD data is being generated, and there is also a method of performing only a search process offline using the generated CAD data. In any method, it takes a very long time to search the entire area. For example, if the target space is all over Japan and the current target area is Tokyo, only the block address in Tokyo is searched. Try to narrow down by the method.

  In addition, the forty-first means searches for a block in the direction of the direction of the camera 10. That is, a block in the visual field 10V of the camera 10 is a search target. Here, the CAD data to be generated is classified and managed by block addresses at the time of generation, so that the target CAD data can be automatically narrowed down if a block is designated. If processing is performed by specifying a block, the search can be performed efficiently.

  FIG. 34 is a block diagram of a visual information processing apparatus which is an embodiment of the forty-second means. This visual information processing apparatus has a configuration in which the image information processing apparatus 100 is provided with two systems for inputting a search command signal from another CPU 300 and two systems for outputting a search result as an I / F. Of course, the number may be three or more, and may be configured to be connected to a plurality of CPUs 300 instead of a single CPU 300.

  As described above, by providing the image information processing apparatus 100 with a plurality of I / Fs, even if the processing speed of the image information processing apparatus 100 is increased, the data transfer speed with the CPU 300 is slow, and the improvement of the processing capability of the system is limited. Such a situation can be avoided. The CAD data may be composed of many points and line segment data, or a lot of candidate data is searched by searching, and color or shape information is read in addition to the position and orientation information. Takes a lot of time. Therefore, by providing a plurality of I / F entrances in the image information processing apparatus 100, it is possible to efficiently perform a search process for a lot of CAD data.

  With reference to FIGS. 35 and 36, embodiments of the 43rd means to the 50th means will be described.

  FIG. 35 shows the concept of the generated CAD data. Actually, the CAD data is composed of a set of fine line segments and surface data. Here, the 31st means and the 32nd means are used to preliminarily register the search CAD data as one set. Shall. They are specifically desks, vases, flowers, and cups.

  Therefore, since a similar set of CAD data exists when the CAD data in the room is generated, the search is performed using CAD data generated in advance only for the object for searching. For example, if a search is performed using flower data, a line segment or a block of surface data constituting the flower 905 is searched, and this is defined as ID = 1. ID = 1 is a flower in the process. Next, when the vase 906a is searched, the data of that portion is searched, and it is defined as ID = 2. Similarly, the data portion of the cup 906b is defined as ID = 3, and when searching using the data of the desk 907, a group of line segments and surface data which are considered to be a desk are searched, and this is defined as ID = 4. To do. When this ID is registered, the CAD data of the line segments and faces constituting the ID is grouped. In the CAD data defining the ID, the position of the representative point of the ID and the posture information of the object as a cluster may be registered. If the matched portions are defined and managed as a lump in this way, the next search can be conveniently performed by specifying an ID and performing a search.

  If there is a lump of the portion 908 that has not been matched with the CAD data for search prepared in advance, the lump portion is an unrecognized object. In such a case, the application system may move the camera up and down, left and right around the unconfirmed object 908 to photograph the unrecognized object 908 and generate CAD data with higher accuracy. If the unidentified object 908 is managed by attaching a new ID number that has not been taken so far, even if there is an unconfirmed object, it is observed closely and detailed CAD data is generated to create a new one. It can be recognized as an object. Here, the fact that an ID is attached means that the system recognizes the object by the ID number.

  The ID may have a type and an individual number. For example, when there are two cups, ID = (1 of 3) may be managed first, and ID = (2 of 3) may be managed second. This means that they are managed with types and individual numbers.

  The 45th means associates human words with the ID. The concept of CAD data in this case is shown below.

  In this way, ID = 1 can be searched for “flower” by making it correspond to “flower” in human language, and the I / F between the system and human can be operated ( It can be constructed to be easy to instruct). Further, when searching, for example, if there is knowledge that “a cup (ID = 3) or a vase (ID = 2) is often on a desk (ID = 4)”, at the time of searching, First, the CAD data attached on the CAD data of the desk (ID = 4) can be searched for. Thus, it becomes possible to raise search efficiency by specifying and searching a range.

  FIG. 36 shows an embodiment for inputting knowledge data. In the search described with reference to FIG. 35, when a block of CAD data of an unrecognized object 908 occurs, the system The display screen of the LCD when the question “What is this object?” Is output to a human being and the human being inputs that it is defined as ID = 5 is illustrated.

  At this time, if knowledge data is also input together, a system can be constructed that eliminates input leakage. Knowledge data can be easily input by inputting it in human language. For example, after inputting ID = 5 = “parcel”, if knowledge 1 = “parcel” is entered “above” of “desk”, a necessary database is automatically created. If the words of all IDs are registered first, the knowledge database can replace the human words with the ID numbers and store the ID numbers in a predetermined place. Here, the word “above” may be stored by replacing it with some other symbol. However, “being over” means that the object on the next ID (space) It is premised that a search program is described so as to correspond to searching for CAD data at a higher position.

  Forty-eighth means considers the case where the search CAD data is collectively registered and the ID is also registered, and the knowledge data is also prepared in advance in a separate file offline, and the knowledge data is collectively read into the system. The knowledge data can be registered efficiently.

  It should be noted that the concept of the CAD data database is that the data is simply recorded in the record No. in the embodiment described above. Although it is described so that it is enumerated every time, when actually realizing the system, it is necessary to make the database easy to search, shorten the time to access various data, and finally the visual information processing device or It is advisable to increase the processing speed of the system so as to improve the real-time property.

  An embodiment of the 50th means will be described with reference to FIG. Originally, the block of individual CAD data in FIG. 35 is actually a complicated and incomplete collection of generated line segment data and surface data. In addition, since the number of data is large, the 50th means is to replace the portion with a basic figure defined by a basic figure or a set of basic figures when the matching degree matches to some extent after searching. . By replacing all replaceable CAD data with basic graphics in this way, the amount of data can be greatly reduced.

  FIG. 37 is a block diagram showing a basic configuration of a visual information processing system which is an embodiment of the 51st means. The image information processing apparatus 100 is configured to input a movement information signal output from an imaging apparatus movement sensor such as the acceleration sensor 30 and the gyro sensor 40 attached to the imaging apparatus 10 in addition to the image information obtained by the imaging apparatus 10. ing. In this embodiment, the image pickup apparatus 10 includes a zoom lens 12, and the image information processing apparatus 100 performs lens control such as the degree of zoom and focus and iris if necessary, and the lens state is a position signal. As a feedback to the image information processing apparatus 100. If the zoom lens 12 is set to auto focus or auto iris, the necessary lens control signal only needs to be adjusted for the zoom ratio (view angle). If the zoom lens 12 is used, even if an object is photographed from a far away position, the field of view is narrowed, but the resolution can be increased, which is effective when it is desired to accurately measure from a far away position. However, in this case, it is necessary to know the field of view when performing 3D measurement. Therefore, if a position feedback signal is taken in, 3D measurement can be calculated accordingly even if the field of view changes. Of course, since the change in the zoom ratio (view angle) is equivalent to moving the camera closer to or away from the object, the position feedback signal of the lens system should not be captured and tracking of the corresponding point should be performed. It is also possible to calculate the 3D measurement by detecting that the position has changed.

  In the case of image information, since there is a scanning time when shooting, for example, when the imaging device 10 is moved quickly, the feature points and the like tracked by tracking immediately go out of the field of view. Therefore, the tracking process will not be in time. Further, when the movement of the imaging device 10 is further accelerated, an image starts to flow, and normal image processing cannot be performed. In particular, when the orientation of the imaging device 10 is changed, the image scene moves quickly. Although a high-speed shutter may be used to prevent the image from flowing, the 51st means detects the amount of movement when moving quickly using the signals of the acceleration sensor 30 and the gyro sensor 40, and the detection result is Based on this, the position or orientation of the imaging device 10 is estimated. Accordingly, even if the position and orientation of the imaging device 10 are lost in the image processing, if a certain position and orientation are estimated based on signals from the acceleration sensor 30 and the gyro sensor 40, when the movement speed returns to an appropriate level, Since it becomes possible to take correspondence between where the user is looking from, CAD data can be continuously generated.

  Here, even when the imaging apparatus 10 is moving at a low speed, if the object to be imaged is a white wall or the like where no feature points exist, it is normal because there are no feature points for image processing in the image scene. Image processing cannot be performed. Even in such a case, the subsequent processing can be continued by using the movement information obtained from the signals of the acceleration sensor 30 and the gyro sensor 40. The processing in the case where the subsequent processing is continued can be realized by the following procedure.

  A specific process embodiment will be described with reference to FIGS. 38 to 41. FIG. 38 shows the concept of CAD data by the CAD data group 991a generated using the CAD data generation start point of the first imaging apparatus 10 as a coordinate system 992a. Among these, the position of the imaging apparatus 10 when normal continuous processing of image information cannot be performed (becomes uncertain) is from when the coordinate system of the imaging apparatus 10 is in the position and orientation of the coordinate system 992b. . In FIG. 39, when the coordinate system of the imaging apparatus 10 is in the position and orientation of 992c when the image processing is enabled again next time, the coordinate system starts from the coordinate system position 992b where normal image processing cannot be performed. Information on the movement distance and rotation angle up to 992c (Δx, Δy, Δz, Δα, Δβ, Δγ) can be obtained by using the movement information obtained from the movement information signals of the acceleration sensor 30 and the gyro sensor 40 described above. Therefore, it is possible to recognize in the coordinate system of the first coordinate system 992a that the position of the imaging device 10 when restarting is the coordinate system 992c. However, since there is an error in the movement information obtained from the movement information signals of the acceleration sensor 30 and the gyro sensor 40, the image processing suddenly includes information on the movement distance and rotation angle (Δx, Δy, Δz) in the first coordinate system 992a. , [Delta] [alpha], [Delta] [beta], [Delta] [gamma]) are not converted and additionally generated, but at the beginning of the restart, the surrounding CAD data is generated in the first coordinate system 992c that has been restarted. As a result, CAD data 991b in the coordinate system 992c is newly generated in FIG. If the amount of CAD data has been generated to some extent, matching processing is performed so as to overlap the same portion as the CAD data 991a previously generated in the coordinate system 992a. Then, as shown in FIG. 41, since the CAD data that generated the same environmental site is generated in the same way, there is a portion 991c of CAD data that matches with high accuracy. The movement and rotation amounts Δx *, Δy *, Δz *, Δα *, Δβ *, and Δγ * from the coordinate 992b to the coordinate 992c for matching the portions with high accuracy are more correct movement and rotation amounts. When performing the matching process, if the movement information (Δx, Δy, Δz, Δα, Δβ, Δγ) obtained from the movement information signal of the acceleration sensor 30 or the gyro sensor 40 is known first, the amount of movement and rotation is calculated. By searching in the direction in which the correlation is improved when the position is shifted little by little in the vertical and horizontal directions, the movement and the rotation amount when the best matching is finally achieved (when the correlation is good) are more accurate. , Δy *, Δz *, Δα *, Δβ *, Δγ *. Similarly, when the movement information obtained from the movement information signal of the acceleration sensor 30 or the gyro sensor 40 cannot be used, the CAD data is newly generated in the coordinate system 992c, and the CAD data is first generated when the CAD data can be generated to some extent. In this case, the initial start position of the search starts from a position where normal image processing cannot be performed. In this case, as shown in FIG. 40, the search for matching is started on the assumption that the newly generated coordinate system 992c is the same position as the position 992b of the coordinate system that has become unable to perform normal image processing. That is, it is necessary to search for matching of the same part of the CAD data of FIG. 40 and the CAD data of FIG. Also in this case, finally, as shown in FIG. 41, coordinate system movement and rotation correction amounts Δx *, Δy *, Δz *, Δα *, Δβ *, Δγ in which the same CAD data portion 991c coincides with high accuracy. * Will be obtained, but since starting from FIG. 40, various movement amounts and rotation amounts will be tried up and down, left and right, and the movement and rotation amount with the most consistent part will be obtained, so the number of calculations, etc. Therefore, it takes a lot of time to search for matching. Still, if normal processing cannot be performed until it can be performed again, that is, if the moving distance and the rotation amount of the coordinates 992b and 992c are small, the movement information obtained from the movement information signals of the acceleration sensor 30 and the gyro sensor 40 Even if there is no, it is possible to complete the matching process in a relatively short time and continue the process. With the acceleration sensor 30 and the gyro sensor 40, the initial position when starting the continuation process can be searched starting from a position closer to the actual position, so the position and orientation of the imaging device 10 can be generated earlier. This means that it is possible to return to the continuation process after correct | amending correctly in the CAD data. Here, the quantity of the CAD data to be matched does not exceed the CAD data generated in the first coordinate system 992a, but whether or not it was able to return to the correct position if it was not matched with a certain quantity or more. Therefore, when a predetermined amount is determined by a certain amount of object, for example, the number of line segments or the number of surface data, and it is determined that the CAD data in a range equal to or larger than the amount is matched, it is determined that the object has returned to the predetermined position. Then, the CAD data newly generated in the coordinate system 992c is converted into the coordinate system 992a and integrated with the CAD data previously generated in the coordinate system 992a. In the subsequent processing, the coordinate system 992a is processed as a fixed reference coordinate system. Until the predetermined quantity matches, the CAD data may be generated in the coordinate system 992c, and a portion matching the CAD data generated in the coordinate system 992a may be searched.

  If no matching part can be searched at any time, the space generated by the previously generated coordinate system 992a has been lost, and it cannot be restored. In that case, the image information apparatus 100 to the application-side CPU 300 or the like 300 cannot correctly calculate the position of the imaging device 10 in the coordinate system 992a, or a new coordinate system 992c starts to be generated and the previous coordinate system A state such as “Lost coordinate system searching state” in which matching search is performed with CAD data generated in 992a, or “Returned from lost state” after returning to the first coordinate system 992a after matching. It is preferable to output a signal indicating that the application-side CPU 300 performs appropriate control in accordance with the state. Further, in the “lost coordinate system searching state”, the name of the coordinate system that has been newly generated as the fixed coordinate system at that time, in this embodiment, the coordinate system of the coordinate system 992c (for example, When the coordinate system 992a is the No. 1 coordinate system, the coordinate system 992c is the name of the coordinate system expressed by No. 2 of the No. 2 coordinate system, etc. The application-side CPU 300 can search the image information processing apparatus 100 for an object search or a current position inquiry in a newly generated coordinate system even in the “search state”, and control the application based on the answer information. You may be able to do it. Based on this information, the application-side CPU 300 may control the movement of the imaging device 10 so that matching can be performed. You may make it move to the next work action. This is effective as a means for allowing control of the application system to continue even when the position of the imaging device 10 is interrupted once by continuous image processing and the position before the interruption is lost. It is. The number of coordinate systems that are lost and start to be generated is No. 2 is not limited to one. 3, No. There may be a plurality of 4,. In addition, when there are a plurality of similar environments as a result of matching, even if it is determined that the environment has returned, it may actually be located at a different location. In such a case, since there is a contradiction in the subsequent behavior control on the application CPU 300 side, it may be determined that the place where it was restored at that time was an error. Even in such a case, if appropriate control content to be performed next is taken into consideration on the application side, the control of the robot or the like can be appropriately restored depending on the control on the application side.

  In this embodiment, the imaging device movement sensor is the acceleration sensor 30 or the gyro sensor 40 directly attached to the imaging device 10. However, when the imaging device 10 is attached to the tip of the robot arm, the robot arm In the case of a mobile robot or automobile, a movement information signal obtained by converting the distance moved from the amount of rotation of the moving wheel may be substituted, or a GPS is attached to the imaging device. The same processing may be performed based on movement information by GPS.

  An embodiment of the 52nd means will be described with reference to FIG. The embodiment shown in FIG. 42 is a mobile work vehicle, and an arm mechanism 400, a CPU 300, an image information processing device 100, a storage device 200, and an imaging device 10 are added to a mobile carriage 502 including a traveling mechanism 500 and a drive control device 501. This is the installed configuration. The imaging device 10 is attached to the tip of the arm mechanism 400.

  The CPU 300 functions to instruct the drive control device 501 to execute the drive control of the traveling mechanism 500 and the drive control of the arm mechanism 400.

  In the mobile work vehicle having such a configuration, since the CPU 300 controls the traveling mechanism 500 and the arm mechanism 400 by itself, the CPU 300 can determine whether or not the imaging device 10 is moving. By making it possible to input the state signal to the image information processing apparatus 100, the image information processing apparatus 100 can recognize the state (moving or stopped) of the imaging device 10. . In addition, when the operation state of the imaging apparatus 10 is input, the image information processing apparatus 100 determines whether the imaging apparatus 10 is moving or the target object is, for example, when the entire image information is moving. Even if it cannot be determined from the image information alone, it can be easily identified.

  An embodiment of the 53rd means and the 54th means will be described with reference to FIG.

  The scene of the image information 137 shown in FIG. 43 has two trees 137e and 137f at the center, an outside lamp 137c on the right side, and a sun 137a on the upper left. When the sun 137a is photographed, in reality, halation or the like may occur and the trees 137e and 137f may not be processed correctly. The 53rd means assumes such a case, and when the halation part, ie, the part of the state where the luminance signal of image information is saturated, is wide, about the whole range or screen (image information), Processing such as generation of CAD data is not performed. As a result, it is possible to prevent generation of strange CAD data. Further, if the range is limited to some extent, for example, if the halation-affected portion is only in the range of the peripheral region 137b of the sun 137a, only the range is not processed.

  In addition, although the light is reflected in the halation like the external light 137c, when the size is small, the position can also be calculated roughly, so the 54th means is an approximate range. The halation site is also generated as CAD data. In this example, the approximate range 137d is the range of the outside lamp 137c, and the outline of the position can be obtained by 3D measurement. In this way, for example, a car light that is seen in the distance cannot be generated accurately as CAD data by halation, but if the halation part is small, its position can be obtained and the light can be obtained. The position of the car can be generated as CAD data.

  The 55th to 57th means will be described with reference to FIG. In the case of stereo measurement, if the distance between two eyes is known, CAD data can be generated based on the length, but in the case of a monocular, there is no absolute dimensional information. A method for defining dimensions is provided by the 55th to 57th means.

  If the CAD data has an absolute dimension, for example, mm, the application side can save time and effort for conversion each time. In the case of monocular processing, for example, a cube with a known dimension may be put in the environment and a unique mark may be put on it, but the absolute dimension is determined based on that. To do.

  In the scene of the image information 138 illustrated in FIG. 44, the cube 138a at the lower right is a cube whose dimensions are known. This may be a sphere, a rectangular parallelepiped or a more complicated shape as long as it has a known dimension. In addition, by taking an image of the ruler 138b shown in the lower left of the image information 138, and knowing the absolute dimension of a predetermined interval of the scale (at least two points are specified), the CAD data is converted into an absolute dimension. Is possible. The scale on the ruler 138b can also be read by image processing.

  An embodiment of the 58th means will be described with reference to FIG. This embodiment is one of basic product forms of a visual information processing apparatus suitable for implementing the present invention.

  In this embodiment, the image information processing apparatus 100 is configured as a visual guidance chip in which the means (arithmetic processing apparatus) for realizing the image information processing function as described above is implemented as an LSI. Using a CCD camera, the storage device 200 uses a memory such as a smart card, a memory stick, an IC card, and a hard disk alone or in combination, and another CPU 300 uses a CPU on the application side such as a robot controller, An acceleration sensor 30 and a gyro sensor 40 are prepared for expansion.

  The visual guidance chip, which is the image information processing apparatus 100, has an input I / F for inputting image information, commands (commands), search CAD data, sensor signals, and the like, responses to the commands, imaging devices 10, 20 and search. An output I / F that outputs CAD data position and orientation information, a memory I / F that connects the storage device 200, and the like are provided.

  The visual guidance chip (image information processing device) 100 outputs the raw image information output from the CCD cameras (imaging devices) 10 and 20 and the acceleration sensor 30 and the gyro sensor 40 in accordance with a command from the CPU 300 on the application side. Processing to generate CAD data by fetching the sensor signal to be performed, execute image information processing for searching for reference CAD data, report the search result to the CPU 300 on the application side, and store the generated CAD data in the storage device 200. Execute.

  Here, the visual guidance chip that realizes the functions of the image information processing apparatus 100 may be divided into a hard chip unit for CAD data generation and update functions and a hard chip unit for CAD data search functions. In that case, the I / F to the storage device 200 is provided in both the hard chip portion of the CAD data generation and update function and the hard chip portion of the CAD data search function, and is controlled not to read / write the storage device 200 at the same time. With such a visual guidance chip configuration, it is sufficient to use only the hard chip unit for an application that requires only the CAD data generation and update functions. When the CAD data search function is required, the hard chip is used. Therefore, a rational application system can be constructed economically.

  46 to 54, specific image information processing (CAD data generation and update function) executed by the visual guidance chip (image information processing apparatus) 100 according to the embodiment shown in FIG. Will be described in detail.

  46 and 47 are flowcharts of main processing executed by the visual guidance chip 100. FIG. Here, an example will be described in which the four simplest commands that are the minimum necessary are prepared as commands from the CPU 300 on the application side. It is assumed that the command = 0 has a function of clearing the data area by resetting the origin, the command = 1 has a camera position request, the command = 2 has a search command, and the command = 3 has a data registration (filing) function.

  The main process of the visual guidance chip 100 is started by a reset signal after the system is turned on. First, in process (1), the current position and orientation data (Xc, Yc, Zc, αc, βc, γc) of the imaging device (camera) are reset to zero to define the world coordinate system. In the world coordinate system, the camera coordinate system at the start of power-on reset is defined as the world coordinate system. Then, the data storage area of the memory 200 that stores the generated CAD data is cleared to zero.

  In the next process (2), the command input from the CPU 300 on the application side is read. Here, each command is distinguished by four numbers of 0, 1, 2, and 3.

  In the next process (3), it is determined whether or not the command = 0. If Yes, the origin of the process corresponding to the command = 0 in the process (4) is reset, and the CPU 300 on the application side is processed in the process (5). Execute the process to output a response to the origin reset completion.

  If the command is not 0 in the process (3), the process jumps to the process (6). In this process (6), it is determined whether or not the command = 1, and in the case of Yes, a response process for outputting the current position and orientation information of the imaging devices 10 and 20 to the CPU 300 on the application side in the process (7). Execute.

  If the command is not 1 in the process (6), the process jumps to the process (8). In the process (8), it is determined whether or not the command = 3. If Yes, the CAD data generated so far in the process (9) is attached with a file name and the generated CAD data storage area of the memory 200 Register in a separate storage area. Then, in a process (10), a process for outputting a data registration completion response to the CPU 300 on the application side is executed.

  If the command is not 3 in the process (8), the process jumps to the process (11). Process (11) is capturing of raw image information from the imaging devices 10 and 20. Raw image information (1) and (2) obtained by the imaging device 10 and the imaging device 20 is taken into the visual guidance chip 100. When the imaging device 20 is not connected, monocular processing is performed. When the imaging device 20 is connected, stereo processing is also possible.

  In the next processing (12), 3D coordinate calculation and CAD data generation processing are performed. Details of this process (12) will be described later with reference to FIG.

  In the next process (13), it is determined whether or not the command = 2. If Yes, the search process for 3D-CAD data is executed in the process (14).

  Then, in the process (15), it is determined whether or not the search has been made. In the case of Yes, the number of candidates, the coordinate posture data of each candidate object, and the candidates are determined as the search results in the process (16). A process of outputting a matching degree and the like to the CPU 300 on the application side is executed, and the process returns to the process (2).

  If it is determined in the process (13) that the command is not 2 or if the search cannot be performed in the process (15), the process returns to the process (2).

  By enabling the visual guidance chip 100 to execute such processing, various applications can be easily realized.

  FIG. 48 is a detailed flowchart of 3D coordinate calculation and CAD data generation processing, which is processing (12) in the main processing of the visual guidance chip 100 shown in FIG.

  In this process, in process (1), it is determined whether it is stereo or monocular, and in the case of stereo, the process branches to process (2).

  In the process (2), a plurality of corresponding feature points are extracted from the left and right image data of the stereo view.

  In the process (3), corresponding feature points of the extracted left and right image data are converted into 3D coordinates.

  In process (4), 3D-CAD data generation processing is performed. Since a method for calculating 3D coordinates in stereo view is common, detailed description thereof is omitted here, but details of the process (4) after the 3D coordinates are calculated are separately shown in FIGS. 49 to 52. Reference will be made later. Thereafter, the process (5) is executed by joining with the case of a single eye.

  The process (5) is a process that branches in the case of a single eye in the determination of the process (1). In the case of stereo, after executing the processes (2) to (4), the process merges with the process (5). In this process (5), a plurality of feature points or feature objects of the captured image data are extracted.

  Next, in process (6), the feature points that can be associated with the feature points of the previous captured image data are given the same names as the previous ones, the new feature points are given new names, Memorize temporarily. Processes (5) and (6) are feature point tracking processes. In the case of stereo, processing may be performed on at least one of the left and right image information.

  Next, in the process (7), it is identified whether or not a plurality of pieces of image data stored while tracking are prepared.

  In the case of Yes in the process (7), 3D coordinate calculation in the photo measurement of the process (8) is executed. Specifically, a plurality of corresponding feature points are converted into 3D coordinates. At this time, since the position and orientation information of the camera that captured each image is also calculated, the information is also temporarily stored. The camera position and orientation information of the latest captured image is set in variables Xc, Yc, Zc, αc, βc, and γc.

  Then, in the process (9), the 3D-CAD data generation process is executed, and the process returns with RETURN. Process (9) is the same as process (4).

  If a plurality of pieces of image data are not found in the determination of the process (7), the process returns as it is with RETURN. Although the minimum number of image data required for photo measurement is two, it is preferable to measure the photo using a larger number. In the calculation of 3D coordinates, after the start, 3D coordinates are calculated using the camera coordinate system that captured the first captured image as the world coordinate system.

  49 to 52 are detailed flowcharts of the 3D-CAD data generation processing of processing (4) and processing (9) in the processing shown in FIG. Here, the details of the update function in the most basic first means will also be described.

  If this process is entered, M (MAX) feature points Q1 to QM (MAX) converted into 3D coordinates in the process (1) are converted into feature points P1 to P1 of N CAD data that have already been generated. It is determined whether or not the PN is a feature point at a position sufficiently separated from the line segments L1 to LL of the L CAD data already generated. When there are M feature points that are sufficiently separated, the M feature points are added to the already generated CAD data. That is, a process of adding M points to N CAD data is performed. Here, whether or not the distance is sufficient is determined by determining whether the distance between the Q point and the P point or the shortest distance between the Q point and the L line segment is the maximum error that can be caused when the Q point is calculated and the P error. Compare the sum of the maximum possible errors when the point or the point close to the Q point (LP point) of the L line segment is calculated. Judged to be a new point that was sufficiently far from the point. If determined in this way, the same feature point is not erroneously determined as another feature point. Here, it is assumed that the maximum possible error is simply assumed. However, strictly speaking, the error to be generated varies depending on the direction. Therefore, more accurate determination can be made by considering the direction and the magnitude of the error. So you should do that. This process (1) is a process for determining whether or not the generated CAD data is an update target of already generated CAD data, and M feature points that are not the update target are newly added. is there. The remaining M (MAX) -M feature points become the feature points to be updated, and the update process is performed after the next process (2).

  In the process (2), the feature point that is not sufficiently separated from the existing point P or the line segment and is approaching again is redefined as Q1 to QM for the next process. Here, the number M is M (MAX) -M.

  In the next process (3), KN P points and LP points generated before the feature points redefined as Q1 to QM in process (2) are defined as KP1 to KP (NK). Here, since there may be a plurality of Q points close to one P point, M = KN is not always satisfied, and therefore, the number is defined as a separate number. At this time, since the relationship between the Q point and the KP point that are close to each other is already known, it is noted that if the feature points are close to which feature points, the subsequent processing becomes easier.

  Process (A) from process (4) to process (15) shown in FIG. 50 is a point data update process.

  In the process (4), i = 1 is set, and in the process (14), i = i + 1 in the process (15) until i = KN, and the process after the next process (5) is repeated. That is, the same processing is performed for KN points KP1 to KP (KN).

  In the process (5), KM feature points among the existing points Q1 to QM close to KPi are set as KQ1 to KQ (KM), and in the process (6), TEN = 0 and j = 1 are processed (12 ) Until all the combinations of KQ1 to KQ (KM) are checked with j = j + 1 in the process (13) until j becomes KM.

  In the process (7), the accuracy when KPi and KQj are calculated is compared. If the precision of KQj is UP (improved), the update process is performed in the process (10) or the process (11). If the accuracy is not up, the process jumps to the process (12).

  If the accuracy is increased, for the same KPi point, in the case of the first process, the coordinates of KPi are simply updated to the coordinates of KQj in process (10). For the same KPi feature point, after the second time, the division update process is performed in the process (11). This is because one feature point called KPi is divided into two or more before, so that P point is defined in addition to the first existing KPi point. Therefore, the process of N = N + 1 is also performed here.

  Whether or not KPi is a point (LP point) on the line segment in the process (9) is entered in the determination logic if the feature point on the line segment is not updated in the coordinate value of the process (10). This is because a new feature point needs to be added and divided and updated in the process (11).

  If the above processing is performed for all KPi points, the point data update processing is performed.

  The process (B) from the process (16) to the process (27) shown in FIG. 51 is a process for updating and adding line segment data.

  In the first process (16), when updating the line segment data, a process of picking up all point data in the target range is performed. That is, all point data in the field of view of the image data referred to when converting to 3D coordinates are extracted and defined as KN KP1 to KP (KN) points. Here, the P point added in the previous process and the P point with updated coordinate values are also included.

  Next, i = 1 is set in the process (17), and the processes after the process (18) are repeated while i = i + 1 in the process (27) until i = KN in the process (26). That is, the same processing is performed for all target point data, KN points KP1 to KP (KN).

  In process (18), j = i + 1 is set, and in process (24), j = K + 1 is set, and j = j + 1 is set, and all combinations of KP (i + 1) to KP (KN) are checked. J is set to i + 1 to KN so that the same combination need not be processed twice.

  In the process (19), the next process is divided into a process (20) and a process (22) depending on whether or not the line segment data connecting the points KPi and KPj already exists.

  If the line segment data already exists, it is checked in process (20) whether either of the points KPi or KPj has been updated or added, and if neither has been updated or added, the line segment is The process jumps to the process (24) without performing the update process. Since there is no change in either the start point or the end point of the line segment, it is determined that there is no change to review the line segment in the middle, and the process is omitted. Of course, even if there is no change in the start point and the end point, the line segment in the middle may be better imaged. Therefore, the processing (20) may not be provided, and all the processing may proceed to the processing (21).

  The process (21) is a process for updating the line segment, and is a process for revising the line segment data connecting the KPi point and the KPj point while referring to the image data that has become a line drawing by the contour extraction or the like. Since the KPi point and KPj point calculated in 3D coordinates are associated with each other as a point on the two-dimensional image data used for calculating the KPi point in the image data. A line segment connecting the point and the KPj point is extracted and the line segment data is connected to the three-dimensional line segment data, that is, the three-dimensional point sequence data connecting the KPi point and the KPj point, or the KPi point and the KPj point. The function of a three-dimensional straight line or curve is obtained and the line segment data is reviewed and redefined. Here, if either KPi point or KPj point is added or updated in the process (20), it means that the accuracy of the start point and the end point has been improved, so if the line segment data is also reviewed. The processing is expected to improve accuracy. In addition, even if the existing line segment data is a straight line, if it is impossible to fit to the straight line by revising it when the accuracy condition is improved, the line segment data of the straight line at that point will be Change to line segment data.

  On the other hand, if there is no line segment data connecting the KPi point and the KPj point, the process proceeds to the process (22), and there is a line segment connecting the KPi point and the KPj point with reference to the image data as a line drawing. If there is no connecting line segment, the process jumps to the process (24).

  If there is a line segment to be connected, line data is added in process (23). That is, a line segment connecting the KPi point and the KPj point in the image data is extracted, and the line segment data is converted into three-dimensional line segment data, that is, three-dimensional point sequence data connecting the KPi point and the KPj point, or A line segment data is newly defined by obtaining a function of a three-dimensional straight line or curve connecting the KPi point and the KPj point.

  If the above processing is performed for all combinations of the points KP1 to KP (KN), the line segment data updating process is performed. Here, following the processing (21) and processing (23), the surface data may be updated immediately, but in this embodiment, the surface data is updated together in the following processing. ing.

  The process (C) from the process (28) to the process (40) shown in FIG.

  In the first process (28), in updating the surface data, a process of picking up all the line segment data updated in the previous process is performed. That is, all the line segment data updated or added in the process (B) is set as KL1 to KL (KN) as KN line segment data.

  Next, i = 1 is set in the process (29), and the processes after the process (30) are repeated while i = i + 1 in the process (40) until i = KN in the process (39).

  In the process (30), all the surface data whose components are the line segment data KLi are extracted and set as KM surface data KS1 to KS (KM). Here, all the surface data to be extracted are all. All the surface data whose components are the updated or added line segment data KLi are extracted from all the surface data. As a specific extraction method, the line segment data connected to the line segment data KLi is searched on the CAD data, and further, the line segment data is searched such as the line segment connected to the line segment to obtain the first line segment data. A process for extracting all the routes returning to KLi, that is, the closed loop created by the line segment data as a surface, is executed.

  In the process (31), j = 1 is set, and in the process (37), j = + 1 is set until j = KM, and the processes after the process (32) are repeated, and all the extracted surface data KS1 are processed. The same process is repeated for ˜KS (MK).

  In the process (32), it is determined whether or not the surface data corresponding to the surface data KSj is already defined as CAD data. Usually, the CAD data may be managed as the back and front of the same surface with respect to the so-called back and front of the surface composed of the same line segment connection, but here the table is a table with a more detailed shape in the future. Considering that the back side is divided and generated on the back side, the back side and the front side should be extracted as separate closed loops (different surface data). In the process (30), all the surface data related to the newly added / updated line segment data (connection of closed loop line segment data) is extracted, so that the surface data for which the surface data is already defined is also extracted. New surface data that has not been defined is also extracted. Therefore, in the process (32), whether the extracted surface data KSj has already been defined as surface data, or has not been defined so far and has been extracted as surface data for the first time, that is, new line segment data is newly created. It is determined whether or not a newly defined closed loop of line segment data has occurred due to the addition or update. Here, the determination is made by comparing the surface KSj with all the previously generated surface data existing in the vicinity thereof, the maximum error assumed for the line segment at the line segment level constituting the surface, and The maximum error assumed for the line segment constituting the surface KSj is considered, and the determination is made based on whether or not they match within the error. A simple comparison based only on the connection relationship of the line segments that make up the surface shows that when one line segment is updated to two line segments that are divided in the middle, the same surface is shown. However, since the constituent line segments are divided and defined in more detail, it may not be possible to distinguish them from the same plane. In addition, since one surface may be divided by generating new line segment data that divides the surface, all or part of the surface data KSj matches all or part of the existing surface data. Whether or not to match the existing surface is determined. Therefore, it is determined whether there is the same surface data that overlaps within the error. If the surface data already exists as a result of the determination, the process proceeds to the updating process (33), and the surface data corresponding to the surface data KSj is not included in the CAD data created previously. In the process (35), the process proceeds to a process of adding and registering new surface data (line segment connection closed loop).

  In the update of the plane data in the process (33), the point data and line segment data have already been updated. Therefore, the line segment data constituting the plane data may be reviewed while referring to the updated contents. Although it is good, it is also possible to refer to the point data and line segment data before update, which are constituted by the surface data before update. When updating the surface, it is premised that there is existing surface data that matches the surface data KSj. However, when the surface data KSj and the existing surface data are entirely consistent within an error range. Can be defined by replacing each line segment data with new line segment data generated more accurately. If the surface data KSj is smaller than the existing surface data and matches a part of the existing surface data, the surface data of the matching part is newly redefined and the remaining part is replaced with another one. Define and divide as one plane data. Further, although there is a portion where the surface data KSj matches the existing surface data, if there is a portion larger than the existing surface data, the surface data KSj is not updated. This occurs because the extracted surface data includes a case where a large closed loop is extracted, but since a small closed loop surface is also extracted, the update process is performed for small surface data. If this is done, the surface data of that part will be updated. Since CAD data is updated when it becomes more accurate, processing that does not greatly integrate small closed loops is not performed.

  Whether the attribute of the surface data is updated after the line segment constituting the surface data is updated in the process (33). Whether there is sufficient image information to update the attribute of the surface data in the process (34). If not, jump to step (37). If there is updateable image information, the process jumps to processing (36).

  Since the process (35) is a case where there is no surface data generated before overlapping the surface data KSj, the surface data KSj is additionally defined in the CAD data as new surface data. The definition here is the definition of the line segment data constituting the surface data.

  Then, in the process (36), the attributes of the plane data are defined. The attribute of the surface data refers to the image data. Originally, surface data is defined by converting 2D image data into 3D coordinates, so 3D surface data composed of 3D coordinate point data and line segment data is included in 2D image information. Since correspondence with which point corresponds is taken, attributes of surface data such as color and brightness of the portion corresponding to the surface are read from the image information and registered as attribute data of the surface data. When jumping from the process (34), it is an update of the plane data, and therefore the attribute of the plane data originally defined is reviewed and updated. Assuming that it was originally updated in the process (33), the line segment data of the surface outline has been updated more accurately, so that the image information this time represents the correct color and brightness of the surface. In the processing (34), it is judged simply by the size or brightness of the area of the image information of the surface, that is, the outline is updated with high accuracy, but when viewed from an extremely oblique angle. , It may be possible to judge only whether or not the scene is shaded and the outline is clearly visible, but the line connecting the camera and the face at the time of shooting depends on the normal of the face. In the case of being close, that is, in a condition where the surface is viewed from the front, it may be determined more strictly so as to be updated. In addition, when the transmittance and reflectance are defined as the attributes of a surface, the brightness of the object reflected in the surface or the object reflected and reflected in the surface is the position of the object. Since it is obtained after conversion into transmittance and reflectance using information such as relationships, it is not always necessary to define it in the first process. For example, when there is an object that can be seen through, the brightness of the surface data of the object when directly looking at the object, the brightness when viewed through the surface, and the distance relationship between them are all obtained. Sometimes, it becomes possible to estimate and calculate the transmittance of the surface, and it is preferable to update it at that time.

  The above processing is performed for all the surface data related to KLi line segments of KS1 to KS (KM), and further to all line segment data KL1 to KL (KN) that have been updated and added. To.

  FIG. 53 shows a concept when the point data and the line segment data are particularly updated in the 3D-CAD data generation processing described with reference to FIGS. 49 to 52.

  The diagram shown in step (1) is the concept of CAD data before update, and is data called a line segment L1 connecting the points P1 and P2. The dotted circles around the points P1 and P2 are circles of the maximum error assumed when the respective points are measured in 3D, and it is assumed that there is an error circle as a sphere on 3D. An error circle on each line segment is defined in the line segment L1, and a sphere having a radius as the maximum error is defined on 3D. Considering this in all points on the line segment L1, the envelope of the sphere is defined. It can be regarded as a face. That is the dotted line around the line segment L1. Here, the size of the error radius in the middle of the line segment L1 is assumed to be a size that is obtained proportionally by estimation from the error radii at the locations of P1 and P2 at the ends of the line segment L1.

  Step (2) shows a concept in which the newly obtained 3D-converted points Q1 to Q4 are drawn on the CAD data before update. A sphere having a radius as an error when each point is calculated is also drawn with dotted lines at the points Q1 to Q4. Considering the error radius, the points Q1 and Q3 may exist within the error radius of the point P1. That is, the point that was P1 before the update may be Q1, or may be Q3. Similarly, it can be seen that the point Q2 may be the point P2, and the point Q4 may be the point LP1 on the line segment L1.

  Here, the update process in the present invention will be described. In step (3), the maximum assumed when each of the pre-update point data and the newly calculated point data which may be the same is calculated. Compare which of the errors is large (here, compare which of the spheres whose radius is the error is small). If it is small, it can be considered that the measurement is more accurate. Update it. Step (3) shows the concept of the update. That is, when the error circle of the point P1 and the error circle of the point Q1 are compared, the error circle of the point Q1 is smaller, so the coordinate value of the point Q1 is replaced with the point data of the point P1. Similarly, the error circle of the point Q3 is also replaced because it is smaller than the error of the point P1, but since the coordinate value of the point P1 has already been updated to the coordinate value of the point Q1, here a new point P3 is defined and the point Enter the coordinates of Q3. This means that the point P1 before the update is divided into the points P1 and P3 after the update. Similarly, the point P2 is updated by replacing the coordinates of the point P2 with the coordinates of the point Q2, since the error radius of the point Q2 is smaller. Similarly, since the error radius of the point LP1 and the error radius of the point Q4 are smaller at the point Q4, the coordinates of the point Q4 are the coordinates of the point LP1, but the point LP1 is a virtual point on the line segment L1. For this reason, a new point P4 is defined as the CAD data, and the update process is performed by putting the coordinate value of the point Q4 into the coordinate.

  As a result, as shown in step (4), P1, P2, P3, and P4 are updated as point data as updated CAD data. In addition, as for the line segment update, the line segment data L1 connecting the point P1 and the point P2 of the CAD data before the update shown in step (1) is the line segment L2 and the point P4 connecting the point P2 and the point P4. And the line segment L3 connecting the point P1 and the line L1 connecting the point P4 and the point P3 are updated, and these processes are realized by executing the processes shown in FIGS. .

  FIG. 54 is a diagram in which the concept of new data generation and update in the 3D-CAD data generation processing described with reference to FIGS. 49 to 52 is applied to the general graphic generation and update concept. It is.

  (1) shows that the line segment L1 connecting the point P1 and the point P2 has a length of 1 m at first, but the position of the points P1 and P2 is updated more accurately by the next measurement. The concept of updating the length of the line segment L1 to 0.97 m is also shown. This is an update process with increased accuracy.

  (2) shows a situation in which the line segment K2 is further visible while the line segment L1 is generated, and the line segment L1 is additionally generated. This is because the line segment L2 that was initially hidden behind the object and cannot be seen becomes visible when the imaging devices 10 and 20 are moved, and points P3 and P4 are newly generated. L2 also indicates the generated situation. This corresponds to the concept of generating a new line segment.

  In (3), the line segment L1 such as the side surface of the container is visible, and the line segment L1 is also generated in the CAD data. It can be seen that the line segment forms a rectangle S1 and is attached to the first line segment L1. In this case, the line segment of the rectangle S1 is newly generated by making it possible to see what has not been seen so far, and therefore corresponds to the generation concept, but is in contact with the line segment L1 of the rectangle S1. Since the line segment L1 is divided into the line segment L1, the line segment L2, and the line segment L3, the concept is that the line segment L1 is updated.

  In (4), a line segment L1 is initially generated, but when viewed close, the point P1 appears to be divided into points P1 and P3, and the point P2 appears to be divided into points P2 and P4. Since the line segment L1 is divided and generated into the line segment L1 and the line segment L2, in this case, the accuracy is also improved, and this is the concept of division update.

  In the last (5), the situation is such that the side is like the side of the container consisting of the line segments L1, L2, L3, and L4, and the one like the rectangle S1 and the line segment L5 are attached. Also in this case, when the imaging devices 10 and 20 are close to the object and photographed in detail, the left side of the rectangle S1 has a shape of a rectangle S2, and the line segment L5 also becomes a rectangle S3. Also in this case, the update process is performed.

  These cases (1) to (5) are typical examples. However, when 3D-CAD data generation processing is performed based on the processing flow shown in FIGS. 49 to 52, such generation and updating are performed. Will be done.

  An embodiment of the 59th means will be described with reference to FIG. This embodiment also shows a product form suitable for implementing the present invention in another application system. In this embodiment, the image information processing function of the image information processing apparatus 100 and the control processing function of the control CPU 300 on the application side are installed in a software (program) built in the main control device 600 such as one personal computer constituting the remote control device. ) Is a mobile work vehicle realized by executing.

  In the mobile work vehicle in this embodiment, the arm mechanism 400, the imaging device 10, and the communication device 503 are mounted on a mobile carriage 502 including a traveling mechanism 500 and a drive control device 501. The imaging device 10 is provided by being attached to the tip of the arm mechanism 400.

  The main control device 600 of the remote control device exchanges control commands / control states and image information with the communication device 503 of the mobile carriage 500 via the communication device 601.

  The communication device 503 mounted on the movable carriage 502 transmits the received control command to the drive control device 501 to control the traveling mechanism 500 and the arm mechanism 400, and the control state and image information output from the imaging device 10 are communicated with the communication device. 601 is transmitted.

  According to such a mobile work vehicle, the image information processing function software 100 of the main control device 600 takes in image information output from the imaging device 10 that moves together with the mobile carriage 502 via the communication devices 503 and 601. CAD data is generated and stored (registered) in the storage device 200 such as a hard disk. The control function software 300 acquires the position and orientation information of the object in the environment in which the movable carriage 502 is moving, and the position and orientation information of the imaging device 10 from the image information processing function software 100. Referring to this, it is determined how to move the arm mechanism 400 and the movable carriage 502, and a control command signal is given.

  If the image information processing apparatus 100 is configured by software as in this embodiment, it can be built with an I / F that is most suitable for a specific application. Further, by configuring the personal computer (main control device) 600 prepared for executing the application control function software so that the image information processing function software 100 is also executed, the dedicated hardware can be simplified. it can. This is particularly effective as an embodiment in which the operating speeds of the arm mechanism 400 and the traveling mechanism 500 are relatively slow and the processing speed by software does not cause a problem.

  An embodiment of the 60th means will be described with reference to FIG. Here, it is assumed that the object to be photographed is a skate player rotating at a high speed.

  As the imaging device 10 for photographing the skate player 909 rotating at high speed, four imaging devices 10 (1), 10 (2), 10 (3), and 10 (4) are provided so as to surround a place where the skate player 909 slides. Deploy. These imaging devices 10 (1) to 10 (4) may be fixed. The image information processing apparatus 100 (not shown) considers that the image information output from these four imaging apparatuses 10 (1) to 10 (4) is sequentially switched and input directly or indirectly. The image information processing apparatus 100 first inputs the image information of the imaging device 10 (1), then switches to the imaging device 10 (2) and inputs the image information, and further, the imaging devices 10 (3), 10 The image information is input in the order of (4), and then the image information is input while switching to the image information of the adjacent image capturing apparatus in the reverse order such as the image capturing apparatuses 10 (3) and 10 (2). To be configured. If you can install a lot of imaging devices 10 so that the entire circumference of the skate rink (sliding place) can be tightly surrounded, you should switch and input sequentially so that it circulates in one direction. Also good.

  If a plurality of imaging devices fixed in this way are prepared and switched, the switching can be instantaneously performed because the switching is an electrical signal of image information. Assuming that shooting is performed, for example, one imaging device 10 instantaneously moves from the position of the imaging device 10 (1) to the position of the imaging device 10 (2). With this configuration, the present invention can be effectively applied to a shooting target object that moves at high speed.

  Here, it is preferable that the imaging devices 10 are arranged densely so as not to hinder the tracking of image information. Then, CAD data generation processing can be realized without worrying about the installation position of the imaging device 10 during 3D measurement. Further, if the arrangement position of the imaging device 10 is known in advance, the processing in the image information processing apparatus 100 can be simplified by using the position information in 3D measurement.

  An embodiment of the 61st means and the 62nd means will be described with reference to FIG. This embodiment is a manipulator 142f that is mounted on a traveling crane 142e and is attached to the lower end of a mast that can be moved up and down provided on a traveling and traversing cart. The hand of the manipulator 142f has a hand portion, and a normal gripping hand is attached, or in some cases, the gripping hand at the tip is automatically exchanged with an end effector dedicated to detaching a bolt when necessary. It may have a function.

  The normal control of the manipulator 142f is configured such that when the operator inputs a command to the operator console 142b, the control device 142c executes a certain degree of automatic operation. The operation state of the manipulator 142f is configured to be displayed on the LCD monitor 142a, and the operator can check the state of the manipulator 142f by observing the LCD monitor 142a. In addition to automatic operation, the operator can perform manual operation while viewing the LCD monitor 142a. The monitor 142a may be a CRT monitor.

  When the present invention is applied to this system, an imaging device (hand camera) 10 is provided at the hand of a manipulator 142f, and the image information is input to the image information processing device 100 via a line such as a wireless modem 142d. The image information processing apparatus 100 is configured to be connected to the storage device 200 that stores the generated as-built CAD data and the autonomous control device 300. Here, the autonomous control device 300 has a function of automatically generating an automatic program by a specific combination of operation tasks based on a general command from an operator. For example, when the general command “Grip the OX device at the entrance of the building and carry it on the work table in the back of the building” is input, the autonomous control device 300 performs the entrance and work of the building held in advance. By referring to position data such as a table, route (trajectory) information for moving the traveling crane 142e and the manipulator 142f from the current position to the building entrance while avoiding obstacles is generated, and the route information is used as the manipulator 142f and the crane 142e. To the control device 142c. The autonomous control device 300 refers to the as-built CAD data stored in the storage device 200 when generating the route information.

  After that, the traveling crane 142e and the manipulator 142f are moved along the generated trajectory and controlled by the control device 142c until the target OX device is gripped, but the path information is only the originally planned trajectory, Autonomous automatic driving cannot be carried out smoothly unless it is possible to automatically check whether there is a real collision in the environment at that time. Therefore, the image information processing apparatus 100 is configured to generate the CAD data of the environment based on the image information output from the hand camera 10 and update the CAD data of the storage device 200 in real time. The storage device 200 was always checked to see if the orbit was as planned, and if the environment changed, the trajectory was corrected immediately and corrected if necessary. The trajectory (route information) is output to the control device 142c. If the correction is not in time, or if an appropriate trajectory correction program is not prepared for the corresponding situation, it is displayed on the LCD monitor 142a that it is difficult to continue the autonomous control. To notify the operator by issuing an alarm.

  Further, when gripping the OX device, if the position of the manipulator 142f deviates from the planned position due to a stop control error of the traveling crane 142e or the like, there is a situation in which the OX device cannot be reliably gripped by normal automatic control operation. appear. In order to avoid the occurrence of such a situation, the image information processing apparatus 100 accurately detects the position and orientation of the X device based on the image information output from the hand camera 10, and the position and orientation of the hand camera 10. Are configured so that the information can be accurately detected and the information can be given to the autonomous control device 300. In this way, the autonomous control device 300 can know the deviation of the relative positional relationship between the hand and the X device, and therefore, by giving an appropriate control signal to the control device 142c so as to eliminate the deviation. The manipulator 142f can reliably hold the OX device. After that, it is possible to return to the back of the building and transport and install the OX apparatus at the target location while correcting the predetermined location (position) based on the image information of the hand camera 10 in the same manner.

  Thus, when a series of work is completed, the fact is displayed on the LCD monitor 142a to notify the operator that the given work has been completed.

  The above control function is mainly guidance control of the manipulator 142 based on image information during automatic operation. In addition to this, first, when the environment does not change so much, the movement trajectory is planned offline beforehand. You may do it. At that time, an as-built environment is prepared by using the CAD data 200 of the storage device 200, and a CAD model of a crane or a manipulator is also put therein, and these are manually operated and controlled in an off-line state, and an operation locus at that time is obtained. It is also effective to use as-built CAD data for the off-line teaching function that makes the operation locus during automatic operation. In addition, it is possible to confirm that there is no mistake in advance by preparing a function for simulating offline whether or not the correct data has been actually created after offline teaching. Even at this time, more accurate simulation can be achieved by using as-built CAD data.

  In addition, a safety monitoring function is also provided in real time to automatically check during automatic operation that safe automatic operation is performed avoiding obstacles by referring to as-built CAD data that is always generated during automatic operation. If there is as-built CAD data to be generated, it can be easily realized.

  Further, when a blind spot or the like occurs only with the hand camera 10, a camera is also installed at the shoulder position of the manipulator 142f, a camera is also installed at the crane, and further, the floor, wall, ceiling, etc. of the building. It may be configured to generate as-built CAD data using image information from a swing-type fixed camera or the like installed in the camera. In addition, these cameras are preferably configured to generate as-built CAD data by using a zoom lens with position feedback so that accurate image information can be output even from a remote location. If it is difficult to use a zoom lens with position feedback, use a pair of a wide-angle lens camera and a telephoto lens camera with a known angle of view, and need both image information. You may comprise so that it may switch and input according to it. Moreover, you may comprise so that both image information may be processed simultaneously.

  Further, in a manipulator system having a hand at the tip of a telescopic arm that moves up and down with a special manipulator, a similar system can be easily constructed if a camera that looks down is attached to the tip of the telescopic arm. The tip camera may have a swing mechanism as long as there is position feedback. Then, the field of view of the camera becomes wider. Needless to say, a high resolution camera and a fisheye lens may be combined to first correct distortion of image information and then be applied in a wide field of view.

  FIG. 58 shows an embodiment of an underwater ROV constructed by applying the 62nd means. Since the ROV that is deeply submerged in the water may not be visible from the surface of the water, a display function for displaying where the underwater ROV is now is desired. The image information processing apparatus 100 of the present invention can easily realize such a display function. Such an underwater ROV is convenient when used for visual inspection or UT inspection of structures in the sea, lakes, or pools in a plant.

  The underwater ROV 143a can be remotely operated from the operator console (operation unit) 143b in front / rear, up / down, left / right, and left / right turning in the water. Usually, remote operation is often performed while watching a video based on an image information signal from a TV camera provided in the underwater ROV 143a. However, when there are many similar structures 143c in the water, it is difficult to know which structure is being viewed, so a means for confirming the current position of the underwater ROV 143a is desired. The image information processing apparatus 100 of the present invention manipulates image information obtained by the camera of the underwater ROV 143a to generate and display CAD data of the surrounding environment so as to move the underwater ROV 143a to a target inspection location. It is suitable for supporting.

  The image information processing apparatus 100 (not shown) is built in the operator console 143b, and CAD data generated based on image information input from the camera of the underwater ROV 143a and the current position of the camera are displayed on the LCD monitor of the operator console 143b. If it comprises, the position and attitude | position of the camera mounted in structure 143c to be examined and underwater ROV 143a can be visually recognized on operator console 143b, and remote control becomes easy.

  Also, if a drawing of the structure is prepared in advance, the absolute position can be easily associated by comparing it with the generated CAD data.

  Although this embodiment is an underwater ROV, even in the case of an airship, CAD data generated based on image information from an on-board camera that captures the current flight location and the position of the camera If the system is configured to display together, it is possible to easily construct a system for displaying the current position of the airship.

  FIG. 59 shows an embodiment of an automatic cleaner constituted by applying the 62nd means.

  The automatic cleaner 144c in this embodiment includes a camera 10 (which may be two eyes) and an image information processing apparatus 100 (not shown), and obstacles 144a and 144b photographed by moving around in a room and surroundings Generate as-built CAD data based on image information such as walls. Since the CAD data to be generated is three-dimensional environmental data, the upper unevenness information may be actively used to perform dusting control. However, if the floor is to be cleaned, 3 What is necessary is just to comprise so that dimension CAD data may be controlled with the information of the top view seen from the top.

  The automatic cleaner 144c keeps track of the movement by monitoring the route that it has moved around as the current position information of the camera, so what part of the room is currently cleaned by cleaning the area 144d and where it remains? It is possible to control the remaining uncleaned parts to be efficiently cleaned while confirming whether they are in the as-built map. Since the vacuum cleaner is used in each home with different floor plans, it is difficult to register the map information of the room in advance, so the function that can generate CAD data in an as-built manner is effective. If data in the same room is kept until reset and it is not necessary to generate CAD data from the beginning every time, the more CAD data is used, the more accurate it is generated. In addition, when the room is changed, the CAD data is also updated, which is also effective in such a case. In addition, if the processing speed is real time, a resident walking in the corridor can also execute control to avoid collision by detecting in real time. Every day, when cleaning is started, it is configured to display a plan view of CAD data that has already been generated, and if you set the vacuum cleaner first, absolute identification of the first position will be It becomes easy. Of course, it is also possible to automatically detect the absolute value by performing partial matching between the CAD data of the surrounding environment and the previously generated CAD data.

  FIG. 60 shows an embodiment of a character reading apparatus constructed by applying the 63rd means. Since the image information processing apparatus 100 of the present invention generates three-dimensional CAD data, CAD data such as engraved characters can be easily generated and matched. Here, an example of an apparatus that reads a moving character will be described.

  Since appears to flow characters to be displayed on the electric bulletin board 145b, which is installed in a building 145a, in the normal way, keep in Determine in advance how much of the character comes to move in what timing, lightning the camera bulletin board Although it is necessary to install a dedicated processing program in front of 145b, if the image information processing apparatus 100 of the present invention is applied, CAD data of characters in three dimensions even when the electronic bulletin board 145b is viewed from an oblique direction. Therefore, it is not necessary to install the cameras 10 and 20 in front of the electric bulletin board 145b. In the search, if a character closer to the basic character such as Mincho and Gothic is searched, it is not necessary to perform an operation of checking and registering a character unique to the electronic bulletin board in advance. Furthermore, since the time is entered in the generated CAD data, for example, it is easy to obtain the moving speed from the time and the moving distance. From the electronic bulletin board 145b, from 20 seconds to 28 seconds at 10:15. If the generated CAD data is processed so as to be connected, it is possible to read a sentence that is not displayed in full text at a time in the form of a full text such as “Today is sunny and cloudy.”

  FIG. 61 shows an embodiment of a three-dimensional measuring apparatus configured by applying the 64th means to the 66th means. If the camera is brought close to the object to be photographed, the resolution, that is, the measurement accuracy is improved. If the camera is close to the object, three-dimensional measurement is performed based on image information obtained by photographing using a general-purpose camera, A system for generating three-dimensional CAD data can be constructed.

  61 has a camera 10a and an image information processing device 120a connected to each other, and generates and stores CAD data based on image information obtained by photographing the environment around the target object 901a. The structure memorize | stored in the apparatus 200a is shown. At the shooting site, such CAD data generation work is performed, and when it is finished, the image information processing apparatus 120a and the computer 146c are connected so that the CAD data 902a of the storage device 200a of the image information processing apparatus 120a can be taken out. Constitute. The CAD data 902a generated by the image information processing apparatus 120a has a data structure suitable for high-speed processing, and is converted into a general-purpose CAD data structure and output when extracted. If the CAD data 902z output to the computer 146c is converted into a data structure that can be used for other applications for general purposes, the generated CAD data can be easily used. Since CAD data also includes dimension information, the system constructed in this way can be used as a three-dimensional measurement system by checking the coordinates and size (length) of CAD data on site as well. Can do. Further, in the method of generating CAD data, CAD data may be generated while photographing, but for example, image information is recorded on a recording medium 146a such as a tape or MO by using the camera 10z in FIG. This may be input later to the image information processing apparatus 120a to generate CAD data. If CAD data is generated at the shooting site and the content is displayed as a picture of the CAD data, the mistakes in the field work will be reduced, but in narrow spaces where monitors and processing equipment cannot be brought in, A method for generating CAD data is effective. In addition, when the CAD data 902a is converted into CAD data 902z and output, the data is not simply converted from a point to a point, line data to line data, and surface data to surface data. If matching processing is performed on some of the figures and converted into basic figures for output, CAD data that is easier to use can be extracted. Although there is a method of performing the conversion function to the basic figure by the function of the image information processing apparatus 120a, a system may be constructed in which another conversion software is prepared in the computer 146c.

  An embodiment of the 67th means will be described with reference to FIG. In this embodiment, CAD data of the environment is generated. The 67th means is a means suitable for generating townscape CAD data, that is, three-dimensional map data. The camera is mounted on an automobile or a two-wheeled vehicle, or the environment is photographed while a person moves the camera. This is means for generating CAD data based on the image usage. The embodiment shown in FIG. 62 is configured to generate CAD data by inputting image information obtained by photographing a building in a town with a camera 147a mounted on an automobile to the image information processing apparatus 100 (not shown). . In the sky, a camera 147b is mounted on the lower part of the airship, and an image such as a building below the sky is photographed to obtain image information. Similarly, the image information of the camera 147b is input to the image information processing apparatus to generate CAD data, and both CAD data are combined.

  Although it is possible to generate CAD data below the streets using only the image information obtained by the camera 147a mounted on the car, the CAD data is generated from the bottom, especially the upper surface of the building. Can not do it. On the other hand, CAD data cannot be generated only by image information obtained by photographing with the camera 147b mounted on the upper airship because the lower, particularly the lower surface, is a blind spot from above. By integrating CAD data generated based on both pieces of image information, accurate CAD data from the bottom to the top of the street can be generated.

  FIG. 63 shows an embodiment of a navigation apparatus constructed by applying the 68th means. The real environment scene 148a is a scene in which a car is approaching an intersection with a traffic signal. It is assumed that this scene is generated in advance as three-dimensional CAD data 148b.

  A traveling car is equipped with a camera (not shown), and image information processing is performed while updating existing CAD data with the latest CAD data generated based on image information obtained by photographing with the camera. Based on the camera position and orientation information obtained by the apparatus 100, the current camera position and orientation information is included in the three-dimensional CAD data 148b (the position and orientation information of the vehicle if the position of the camera attached to the vehicle is known). Is displayed. Thereby, it can be known where the car is currently running in the CAD data 148b. With this configuration, a car navigation device that does not require GPS can be configured. Also, if CAD data is available in advance, when the destination is indicated, the function of displaying and navigating the best route from the current position to the destination is made to function in the same manner as a conventional car navigation device using GPS. Can do. If 3D CAD data is not prepared in advance, if there is at least 2D map information, the 2D map information can be associated with the CAD data generated at that time. Thus, the guide to the destination can be performed on the two-dimensional map, and the current position is confirmed by associating the generated CAD data with the two-dimensional map. The two-dimensional map information should be easily associated with the three-dimensional CAD data generated by providing the point points with the three-dimensional information. For example, the presence or absence of a characteristic building or traffic signal may be prepared with three-dimensional CAD data even if it is a basic figure that simulates an approximate shape. Further, when the same place as the mark prepared in the middle can be searched from the generated CAD data, it is preferable to announce that it has passed there. For example, if a traffic signal in the three-dimensional CAD data 148b comes into the field of view of the camera, it is configured to announce “where has passed an intersection”. Such a case is realized by using the search function of the image information processing apparatus 100 and, here, making the reference CAD data search using the traffic signal data 148d. Further, by using the reference CAD data 148c of the destination HITACHI building, the destination can be searched. In this way, when a thing that serves as a mark of the passing point or a mark of the final destination (in this case, a signboard of the HITACHI building) can be searched, it can be searched on the displayed CAD data 148b. It may be configured to change the color so that it can be seen or display the part in a blinking manner to notify the location. When the vehicle goes to a place where CAD data is not prepared in advance, the vehicle is not only updated, but also actively generates new CAD data. As a result, a wider range of CAD data of azbilt is generated as the automobile runs. If there is no map, navigation to the destination cannot be performed, but new map information is accumulated because CAD data is generated up to the place where it has been running, so the second time From now on, navigation to the destination can be reliably performed using the map information. In addition, it is preferable to provide a function for specifying a predetermined area of the generated CAD data and registering that portion as a search CAD data. The reference data 148c whose registered name is the HITACHI building is registered in this way. When registering, signs with unique shapes, colors and patterns are good, so signboards are appropriate. As a general-purpose mark such as a traffic signal device, several typical types may be registered in advance and registered before being supplied to the user of the car navigation device.

  CAD data is generated while running in a car, but it is difficult to run around in Japan with a single car, so the newly generated CAD data is stored in the central information using the data communication line of a mobile phone. Enable to send to processing center. In this central information processing center, it is possible to create CAD data of roads throughout Japan by receiving and integrating the latest CAD data sent from automobiles in various places. In addition, CAD data created by integration at a central information processing center can be updated using the latest CAD data in the required area using a data communication line such as a mobile phone if requested by the car navigation system of each car. The information can be transmitted from the information processing center to the car navigation device of each car. Of course, the integrated CAD data may be input as 3D map information as data of a car navigation device of each automobile using a storage medium such as a CD.

  When integrating the CAD data, the central information processing center carefully integrates the date and time (time) when the CAD data was generated. In other words, snow may be accumulated in winter data, and CAD data may be different even in the same topography in summer and winter. In addition, since the CAD data generated in the night and the CAD data generated in the day are different in a short cycle, when integrating the CAD data, the CAD data is classified while being classified according to the generation time. Data such as affected snow and night lights should be integrated to be managed as temporary.

  An embodiment of the 69th means will be described with reference to FIG. This embodiment is an LCD monitor in a car navigation device, and is preferably configured to selectively display a plan view and a bird's eye view as necessary. FIG. 64 is a plan view displaying a scene when the vicinity of the intersection where the automobile 149a is approaching is viewed from above. One of the intersecting roads is a narrow road, from which the car 149b is going out to a wide road. A pedestrian 149c is also displayed on the narrow road.

  If such information can be displayed on the LCD monitor of the car navigation device, it is possible to prevent a collision at the intersection. In addition, it is possible to drive with caution regarding pedestrian jumping. The display form should be devised so that the moving direction, speed, etc. are easy to see. For example, the faster the moving speed, the shorter the blinking period and the faster the blinking speed, or the display color is changed from blue to yellow and red, so you can see that the dangerous speed is intuitively understood. It is good to be.

  The car navigation device of the automobile 149b can also transmit information such as the current position and posture of the vehicle (own vehicle) and the moving speed if necessary to the car navigation device of the automobile 149a nearby. The car navigation device of the nearby vehicle 149a is configured to display the information received from the car navigation device of the vehicle 149b on the LCD monitor, so that the risk of a collision between the vehicles 149a and 149b is displayed to the driver. Can be recognized. Also, the pedestrian 149c carries a navigation device having the same function as the car navigation system, and the navigation device of the pedestrian 149c also transmits information such as the current position and orientation of the pedestrian 149c, so By configuring the navigation device to display this information on the reception LCD monitor, the driver of the automobile 149a can recognize the presence of the pedestrian and can predict the possibility of the pedestrian jumping out.

  If each navigation device is configured to send and receive their own information and display them on the LCD monitor of each navigation device, it becomes possible to recognize each other's environmental situation (the state of the other party) and act.

  This embodiment is a configuration in which navigation devices directly exchange information, but each navigation device transmits information such as its position and posture, speed information if necessary, to a central information processing center, The central information processing center is configured to collect and manage all information sent and to distribute and display information of other navigation devices near the navigation device to each navigation device. It is also possible to do.

  In such an embodiment, in a dangerous situation, it may be configured to issue an alarm so as to call attention, and in the case of a car, a dangerous driving operation is linked with a driving control device of the car. It may be configured to implement a function for limiting the above.

  An embodiment of a navigation device carried by a pedestrian will be described with reference to FIG. The navigation device in this embodiment is configured as a navigation device 150a in the form of a hat that a pedestrian wears on the head as necessary, or the navigation devices 150b and 150c in a form of being put on a shoulder. Each navigation device 150a to 150c includes an imaging device (camera) 10, an image information processing device 100, a storage device 200 with a wireless communication function, and a battery (not shown). The whole should be small and light. An LCD monitor is not required for devices intended to protect elderly people and children from traffic accidents, and it has the function of detecting its own position and transmitting it to a central information processing center or a car navigation device of a nearby car. If there is enough. It is desirable that the pedestrian be configured to carry any one of the navigation devices 150a to 150c. If the camera is configured to be placed on both shoulders, it is possible to take a picture of almost the entire circumference without blind spots. In addition, a wearable device may be obtained by baking a camera or a processing circuit on a sheet-like substrate and mounting it on a hat or clothes. The camera and processing circuit may be further miniaturized and configured into a pen shape that can be placed in a breast pocket, or incorporated in a wristwatch, brooch, pendant, glasses, earring, or the like.

  FIG. 66 shows an embodiment of a picking system configured by applying the 70th means. The picking system is a system in which parts placed randomly in a production site or the like are handled one by one by a manipulator 151a.

  The imaging device (TV camera) 10 is attached to the tip of the manipulator 151a, and the positional relationship between the component gripping hand 151b and the TV camera 10 is set in advance to the visual guidance chip constituting the image information processing apparatus 100 as known information.

  A video signal which is image information of the TV camera 10 is input to the image information processing apparatus 100 and processed. The motor 151c of each joint of the manipulator 151a is driven and controlled by the drive device 151d to adjust the position and posture of the gripping hand 151b at the tip by changing the angle of each joint, and a control command for this is given by the manipulator control device 300. The configuration generated by is the same as the configuration of the control system of a general industrial robot.

  The control apparatus 300 is connected to the image information processing apparatus 100 via a communication cable, and acquires information to be referred to when generating a control command from the image information processing apparatus 100. Moreover, the control apparatus 300 acquires a work command from the operator 151e.

For example, when a work command is given such that the part A on the work table is put into the tray A and the part B is put into the tray B, the control command generation unit in the control device 300 performs the following processing. To do.
(1) First, a command to start generation of CAD data on the work table is given to the image information processing apparatus 100.
(2) Next, the manipulator 151a is operated so that the TV camera 10 at the tip can take an image of the work table from various angles. This can be achieved by teaching the operation locus of the manipulator 151b in advance assuming an area that does not contact the part, and controlling the operation to perform a playback operation.
(3) Next, if CAD data 151f on the work table is generated to some extent (this is because even if the update contents of the CAD data are reduced in the processing in the image information processing apparatus 100 and the same operation is repeated, the CAD data 151f is generated. It may be determined by receiving a signal from the image information processing apparatus 100 that the data has been generated so as not to be updated, or it may be determined empirically that the amount of movement of the TV camera 10 is sufficient. However, the image information processing apparatus 100 is instructed to search for the component A. The CAD data 151g for searching for the part A may be registered in the control device 300 and transmitted to the image information processing device 100 together with a command when necessary, or may be registered in advance in the image information processing device 100. It may be possible to specify the part number. The position and orientation information of the TV camera 10 may be constantly transmitted, or may be transmitted at the current timing.
(4) If there is a response of the position and orientation information of the part A on the work table from the image information processing apparatus 100 as a search result of the part A, the manipulator 151a determines which part of the plurality of candidates is to be gripped first. The priority order is determined from the positional relationship with The function for determining the priority order may be provided in the image information processing apparatus 100 so that only the position and orientation information of the first candidate component is sent.
(5) Next, the manipulator 151a is controlled to grip so as to grip the determined part A. Since the position and posture of the part A are known, it is possible to easily plan where and in what posture the gripping hand 151b should be accessed. Then, based on the plan, the control trajectory at the tip is planned and converted into the operation angle of each joint to execute specific control. The position and orientation information of the TV camera 10 may be checked for confirmation. However, when the manipulator 151a is fixed to the movable carriage and may move and the position may be shifted, the position and orientation information of the TV camera 10 is in the space of CAD data for obtaining the positions and orientations of the parts A and B. Therefore, when the manipulator 151a or the work table moves and shifts, the manipulator 151a is controlled while referring to the current position and orientation information of the TV camera 10. Good.
(6) If the gripping hand 151b of the manipulator 151a grips the part A, the part A is transported to the tray A and the part A is put into the tray A. The determination as to whether or not the component A has been gripped may be performed by providing a pressure sensor or the like on the gripping hand 151a and confirming that the gripping hand 151a is securely gripped to move to the next operation control.
(7) With respect to the part B, if the same process is performed with reference to the CAD data for search 151h, the part A and the part B can be automatically classified and accommodated in separate trays.

  Although the example of the grip sensor has been described, the autonomous control type robot is also the same. However, in all application cases, it is not necessary to control only by the information of the image information processing apparatus 100. It is preferable to determine how to control by comprehensively judging information of various sensors.

  An embodiment of an inspection apparatus configured by applying the 71st means and the 72nd means will be described with reference to FIG. FIG. 67 is a schematic diagram of a production line such as a CD and a DVD in which the inspection apparatus according to this embodiment is applied to the inspection process part.

  The DVD 152c that is the object to be inspected is conveyed by the conveyor 152b, and is imaged by the imaging device (camera) 10a provided at the tip of the manipulator 152a. The number of manipulators 152a provided with the camera 10a may be one, or a plurality of manipulators 152a may be installed in order to prevent an inspection omission earlier. The manipulator 152a images the DVD 152c from various angles, inputs the image information to the image information processing apparatus 100, and generates three-dimensional CAD data of the DVD 152c that is the object to be inspected. On the other hand, in the image information processing apparatus 100, CAD data of the DVD 152c which is a normal inspection target object is also registered in advance as reference CAD data, and compared with the generated CAD data to extract different portions. Then, the quality of the DVD 152c that is the inspection object is determined (automatic inspection). For example, a good DVD may be photographed in advance and generated as CAD data, and the search function of the image information processing apparatus 100 may be used, or a matching function may be separately prepared. For example, if there is a scratch 152d on the surface of the DVD 152c, the portion can be extracted as a large difference, so that the DVD is a defective product. It is determined that

  Match the 3D CAD data with the best matching posture, or match the posture so that the reference parts match, and perform differential processing (difference processing is performed by changing each CAD data into a set of point data. )) To extract different parts, and the extracted parts (point data) volume, area, length, etc., different amounts (number of point data), or color differences (point color) It is also possible to make a determination based on the difference result between the information), the area of the portion where the color difference is greater than a predetermined value (number of point data), and the like. Further, if the reference CAD data is ideal CAD data generated from the design drawing, the standard can be made more reliable. Since photographing is performed from various angles as described above, it becomes possible to realize automation of a detailed appearance inspection.

  According to the 72nd means, since the TV camera 10 is equipped with a microscope, finer defects can be generated as three-dimensional AD data, and a detailed inspection can be performed.

  In the above embodiment, the inspection apparatus is a form in which the inspection target object is fixed and the camera is moved by a manipulator or the like. However, the inspection apparatus is configured to capture the image while moving the inspection target object with the camera fixed. May be. Moreover, you may comprise so that both a camera and a test object may be moved. Also, various facial expressions of a human face are photographed and three-dimensional CAD data is generated by the image information processing apparatus 100, so that it is registered in association with the person's ID (name, etc.), and again the same person. You may make it apply to the inspection system which determines whether or not. Further, the present invention may be applied to an inspection system that generates three-dimensional CAD data of a fingerprint in combination with a microscope and confirms the identity of the person using the fingerprint.

  68 to 71 show an embodiment of two home robots configured by applying the 73rd means to the 79th means. An embodiment of the automobile will be described with reference to FIG.

  FIG. 68 is a block diagram of the control system of the pet robot. The control system of the pet robot in this embodiment is a P.I. The image information processing apparatus 100 and the storage device 200 of the present invention are incorporated in the control system of the pet robot AIBO introduced in 53.

  The basic control system of AIBO has a configuration in which the microprocessor 300 controls the movement of the head and the front, rear, left and right legs. In the pet robot in this embodiment, image information obtained by photographing with a camera installed on the head is input to the image information processing apparatus 100 to generate three-dimensional environment CAD data and store it in the storage device 200. To be configured. Then, the microprocessor 300 is configured to be able to refer to the position and orientation information by searching the camera position and orientation information and specific CAD data as necessary. The microprocessor 300 may be configured so that the CAD data generated in the storage device 200 can be directly referred to. Control processing executed by the microprocessor 300 will be described later with reference to FIG.

  FIG. 69 is a schematic diagram of an embodiment of a home robot that performs simple handling. This home robot is a wall-hanging type robot, and includes a gripping hand 154c for gripping a target object and an arm 154a for controlling the position and posture thereof, so that the root of the arm 154a can be firmly fixed to the wall. Constitute. The environment in which this robot is used is assumed to be a home dining room or living room, and is an environment in which a table 154d, a cup 154e, and the like are placed.

  A TV camera 10 is attached to the gripping hand 154c, and image information obtained by photographing with the TV camera 10 is input to the image information processing apparatus 100 and processed, whereby CAD data of the surrounding table 154d and the cup 154e is obtained. The arm 154a and the gripping hand 154c are controlled so as to generate and grip the cup 154e on the table 154d and put it out to the master. A detailed control system of this embodiment will be described later with reference to FIG.

  FIG. 70 is a flowchart of a program executed when the microprocessor 300 in the pet robot described with reference to FIG. 68 performs autonomous control. The contents of the autonomous control described here will be described by taking the case of AIBO as an example, but autonomous control is performed in the same way for the case of an arm in a home robot described with reference to FIG. be able to.

  First, when the power is turned on or the start button is pressed, the microprocessor 300 starts from the start 155a and first executes the process 155b of the first priority task. In the process 155b of the first priority task, the remaining capacity of the battery which is important for the autonomous control type robot is confirmed. If the remaining capacity is small, the location of the charger is searched from the environmental CAD data, The travel route to the place is planned, the vehicle is moved to the charger while avoiding obstacles, and connected to the charger for charging. If a dedicated charger is prepared in advance, the charger is preliminarily registered with features such as shape and color as search CAD data. If charging from a household outlet is possible, register the shape of the household outlet. At this time, if the generation of the environmental CAD data is insufficient and the charger cannot be searched immediately, a process of searching for the charger is performed while moving around and generating the CAD data. If the remaining capacity of the battery is sufficient, the process proceeds to the next second priority task process 155c without executing this task.

  The process 155c of the second priority task is a process of listening to the instruction from the master and executing the contents. The means for recognizing the master as the master may first register the features of the face of the master by the image information processing apparatus 100, or attach a characteristic doll to the finger of the master to recognize the doll. Alternatively, the sound input from the microphone may be recognized as voice and recognized as the master, or may be determined by pressing a predetermined button on the robot body. When an instruction “run toward the master” is issued from the master, the microprocessor 300 uses the image information processing apparatus 100 to search for the position of the master, and from the current position, The movement route to the position is planned, and control is performed along the route while avoiding obstacles. Since the image information processing apparatus 100 always generates azbild CAD data in real time, environmental CAD data is also generated for an obstacle newly generated in the middle. Confirmation is made by referring to the environmental CAD data, and if necessary, the running control is performed while reviewing the contents of the plan of the movement route. If there is a large obstacle on the way and a program to avoid the obstacle is not prepared, or if it is difficult to avoid even if the avoidance program is executed several times, it is impossible to execute for the master. , For example, by telling. Further, when the robot can reach the master as instructed, it is possible to bark with a cry different from the case where the command cannot be executed because the command has been executed. If a condition for executing a task with a higher priority occurs in the middle of the execution task, control is performed so that a process with a higher priority is executed even in the middle.

  If there is no instruction from the master, the process proceeds to the next third priority task process 155d. This third priority task is a task of chasing a suspicious person if a suspicious person is found. The suspicious person is searched for when, for example, an unrecognized moving object that does not apply to the reference CAD data prepared in the past by the image information processing apparatus 100 is detected, or a sound that has not been heard normally is a microphone or the like. When entering, it is recommended to use it as a trigger to start searching for suspicious individuals. When a moving suspicious person is recognized as a cluster of unrecognized objects, the position is detected and moved to follow. If a master appears during this task execution and an instruction is issued, or if the remaining battery capacity is low, the task with the higher priority of the second and first priority tasks is executed with priority. . If the remaining battery capacity is sufficient and there is no master or suspicious person, the process proceeds to the process 155e of the fourth priority task.

  The fourth priority task process 155e is a process of walking around and generating environmental CAD data. In order not to walk indefinitely, it is preferable to control by setting a limiting condition such as within a radius of 5 m. If the CAD data can be generated sufficiently, the process moves to the process 155f of the fifth priority task.

  The fifth priority task process 155f is a process for preventing wasteful consumption of the battery, and detects a change in the surrounding environment by inputting sensor information in a hibernation state, specifically, without walking or standing. To continue the state. In order to save power consumption by the camera and the image information processing apparatus 100, it is preferable to monitor only about a microphone and a contact sensor. Here, what is important is that a task with a high priority is executed whenever the execution conditions are met. The priority may be defined in a predetermined order by a program, or may be set as data in which the priority order can be changed. The change of the priority order is preferably stored by learning so that the master can teach it. In addition, the task processing related to safety is defined by hardware or a program so that it cannot be changed according to the user's preference, and the user can change the priority of the task processing that may be decided according to the user's preference. You may be allowed to.

  The processing described above is an autonomous control type robot mainly for the purpose of moving, but this is the same for automobiles and the like.

The case of an automobile will be described with reference to FIG. Basically, as described in the navigation device, since the best route from the current position to the target position is generated, the vehicle is configured to be controlled along the route. In this case, CAD data is generated in as much detail as possible, not only to distinguish road lanes, but also to generate CAD data of obstacles on the road and other cars parked on the road to avoid it. In consideration of characteristics such as the turning trajectory of the vehicle, the travel route is generated, and the CAD data generated in real time always checks whether the route needs to be changed. It is configured to perform automatic driving while making subtle changes. This can be easily realized by generating and referencing three-dimensional as-built CAD data in real time. Further, in the case of an automobile, a sign or traffic signal around the road is searched, and if it can be found, the contents are confirmed and the automobile is controlled according to the result. For example, when a traffic signal is searched with reference CAD data 148d, information on the color and brightness of the surface is confirmed from the position of the traffic signal lamp, and it can be determined that the blue lamp is lit. Continues automatic driving while avoiding other obstacles. When the red lamp is lit, it automatically detects crossing reports and stop lines and stops it before that. Of course, if there is another vehicle ahead, it will be an obstacle, and if it cannot be avoided to the left or right, it is programmed to perform automatic control to reduce the traveling speed of the vehicle and avoid collision. . In addition, since road signs such as one-way traffic and speed limit are determined in color and size, they are registered in advance as search CAD data. And when always searching for the presence or absence of those signs and searching for those signs, the result is output to the main controller 300 of the automobile, and based on the signs. By performing the control, the vehicle can be automatically operated.

  As a movement command, it is sometimes necessary to control not only the approximate position but also the posture. For example, in the case of AIBO, when performing control to put a body in a narrow place, the target position and posture are finally set in the environmental CAD database so that the posture matches the posture. This can be realized by processing so as to perform control. As a method of automatically setting the position and orientation in the environment CAD database, the master may lift the AIBO to the place where it wants to be taught, and this may be configured to teach the place of XX, or AIBO. Specify the location of the part that is recognized, and determine the expression of the type of posture for the part, such as facing the part, facing the hips, side by side, etc. Then, the configuration may be such that the posture can also be indicated. In the case of an automobile, it may be configured so that it can be set by being placed next to the building at the target location, or by entering an attitude that has entered the parking lot at the target location from behind. When moving the vehicle to the specified posture in the back, the imaging device that captures the rear is enriched, and the newly created obstacles etc. are safely avoided and controlled to become the desired posture and the target position It is configured to be able to move up to. For obstacles on the way, prepare an avoidance program according to those patterns, but for example, if there is no space where the car can pass and it is difficult to avoid with the prepared avoidance program, The automobile is stopped, and it is configured to output a notification that execution is difficult and notify the user (master). Moreover, the priority order in the case of autonomous driving control of a car is preferably as follows.

  1) Safety of pedestrians, 2) Safety of passengers, 3) Safety of passengers of other vehicles, 4) Safety of people in surrounding buildings and structures, 5) Safety of self (own vehicle) (However, priority is often given to passengers.) 6) Safety of other vehicles, 7) Safety of surrounding buildings and buildings, etc. Prepare a program to execute the operation.

  The wall-mounted arm type robot is a work robot such as taking a cup. Therefore, when the work is completed, an end response is returned, and when the work cannot be executed, an unexecutable response is returned. In addition, by controlling a pet robot such as AIBO in the same way as an automobile, it can be applied as a guide dog robot so as to navigate a pedestrian.

  Next, an embodiment of the 76th to 78th means will be described. In this embodiment, a monitoring camera 153a, which is an imaging device independent of the microprocessor 300 that performs autonomous control, is provided in the configuration of the pet robot described with reference to FIG. Image information generated by imaging with the monitoring camera 153a is configured to be transmitted via the wireless system 153b of the independent system and monitored on the master monitor. If the pet robot is within the reach of the master's eyes, the master can take the necessary action in an emergency. In addition, it is only necessary that the pet robot be physically out of the room, but if you can go out of reach of the master's eyes, or enter the garden of another house. In such a case, it is important that the pet robot can always monitor the pet robot under the responsibility of the master because there is a risk that the pet robot may harm other people or destroy things. The monitoring means is not limited to the monitoring camera, but since the monitoring camera can directly capture and convey the scene, it is desirable to provide a monitoring camera so that the situation can be determined correctly.

  In the embodiment of FIG. 68, the surveillance camera 152a is exemplified as being mounted on the back of the pet robot. However, a fixed surveillance camera is installed in a place where the pet robot moves around, and an image of this stationary surveillance camera. A configuration may be adopted in which monitoring is performed using video based on information signals. What is important is that it is an independent system separated from the microprocessor 300 which is an autonomous control device of the pet robot. When the camera of the head of the pet robot is used as the monitoring camera, the head is controlled by the microprocessor 300 and operates in a form necessary for autonomous control. It becomes difficult to make it photograph so that it can see. When the surveillance camera is photographing a completely different place and the master cannot accurately determine the current situation, an emergency stop command is wirelessly transmitted to the control device (microprocessor) 300 at that time. However, since the autonomous control system may be out of control, it is better to be able to send directly to the power-off function of the system independent of the autonomous control system. .

  In the case of a pet robot such as AIBO, the head camera may be substituted, but in the case of a larger humanoid robot, the influence in the event of an emergency will increase, so each place of the body It is recommended that an independent surveillance camera be installed in the system so that in the event of an emergency, a power interruption or emergency stop command can be sent. A robot that is autonomously walking may be more dangerous if it suddenly loses power, so if the master determines that it is dangerous, it launches an emergency stop automatic control program as a top-priority task process. It is desirable to be able to stop autonomously and safely while judging the situation. It is important that this emergency stop program is not affected by normal autonomous control (other tasks). It is also important for the surveillance camera not to be obstructed by normal autonomous control operation.

  In situations where the situation cannot be judged, the emergency stop program will be started at the responsibility of the master, so that the independent surveillance camera can properly capture the surveillance area in any situation. In addition, it is desirable to provide it independently of the autonomous control system. In the pet robot illustrated in FIG. 68, a signal line 153c for notifying the microprocessor 300 that image information is normally transmitted from the wireless device 153b is provided. As a result, when the monitoring video (image information) is not normally output during the autonomous control, the robot can be stopped by operating the emergency stop program. In the flowchart shown in FIG. 70, the process is executed as the process 155g of the task having the highest priority (the 0th priority task) which is higher than the process 155b of the first priority task. Similarly, when an emergency stop command is input as a wireless command from a master watching a video of image information captured by a monitoring camera, first, it may be configured to start this task first. The emergency stop control may be performed by an independent control system that is completely different from the control device 300.

  If the image information is not normally output from the surveillance camera, the surveillance system will be down even if the autonomous control system is normal, so the master will not be able to perform safety monitoring. Configure to stop. When the image information is normally output from the monitoring camera, since the master is monitoring, it can be safely stopped by an emergency stop signal from the master. When the image information of the monitoring camera cannot be displayed normally on the master side, the emergency stop is performed because the monitoring by the master is not possible. The wireless device 153b transmits image information obtained by photographing with the monitoring camera 153a to the master side, confirms a response that the image information has surely arrived from the master side, and uses the line 153c to the microprocessor 300 side. It is possible to take thorough safety measures by composing such that it is normal. After that, it becomes the operation responsibility of the master whether the master is monitoring enough.

  In the case of a car, if the master is on board, the master directly monitors the automatic driving state, and if it is dangerous, respond to an emergency by switching to manual operation or operating the emergency button. The process (a program that decelerates and stops safely) is executed. In addition, an independent monitoring camera may be mounted so that the automatic running state is monitored remotely using information such as car navigation and CAD data, or this monitoring may be automatically monitored. When the master entrusts monitoring work to a third party monitoring agency, the monitoring agency may be responsible for monitoring based on the contract.

  FIG. 71 is a block diagram of a control system of a home robot (arm type robot shown in FIG. 69) which is an embodiment of the 79th means.

  The arm mechanism 156f is a mechanism portion configured by the arm 154a and the gripping hand 154c, and this mechanism portion is configured to be hung on a wall. A control device is also housed in the base case of the arm 154a. This control device includes a control device 156e that controls the operation of the arm mechanism 156f, and how to move the arm 154a to the control device 156e and control the position and posture of the gripping hand 154c at the tip. The autonomous control chip 156d corresponding to the microprocessor 300 that generates the target command is connected. The camera that is the imaging device 10 is attached to the tip of the arm mechanism 156f, and image information obtained by photographing with the camera 10 is input to the image information processing device (visual guidance chip) 100. The autonomous control chip 156d It is configured so that various kinds of environmental data can be inquired to the processing apparatus 100 and the results can be obtained.

  Then, when the power is turned on or the start button is pressed, the autonomous control chip 156d moves the arm mechanism 156f variously to photograph the environment with the tip camera 10 in order to cause the image information processing apparatus 100 to generate CAD data. Thus, it is configured to generate surrounding image information (environment data). At this time, the operation mechanism of the arm mechanism 156f is configured to be considered in operation so that there is no obstacle, or by the control processing by the autonomous control chip 156d, the arm mechanism 156f is initially contracted in a posture in which the camera 10 is contracted. The surrounding environment CAD data is generated in various directions, and the environment CAD data is checked to confirm that there are no obstacles in the surroundings, and the operating range of the arm mechanism 156f is gradually expanded to widen the camera 10 It is good to comprise so that it may move.

  When there is an obstacle that collides within the operation range of the arm mechanism 156f, the arm mechanism 156f is moved so as to avoid it, and images are taken from various angles including the obstacle as close as possible with the camera 10. In this way, the arm mechanism 156f is controlled so as to generate CAD data as accurately and accurately as possible.

  The shape of standard parts such as tables, chairs, step stools, bottles, cans, bags, etc., which are often used in the home, are registered as search CAD data. Tables and chairs have round or square shapes that vary from manufacturer to manufacturer. Search CAD data may be prepared using product samples from all manufacturers, but those with representative shapes are selected. It is good to register it. A master (resident's face or the like) is configured such that a user (master) generates CAD data and registers it first after purchase. In this case, after generating the environmental CAD data, the master stands in front of the newly set arm mechanism 156f, and images are taken from various angles to generate the CAD data, which is registered as the master. Constitute. In this way, since the environmental CAD data has been generated first, the newly added CAD data is an object with a newly registered name, so that it can be easily separated from the environmental CAD data. it can.

  Similarly, the CAD data of individual articles other than the master may be registered. In this case, at first, it is preferable that CAD data for only the room is generated in a state where there is nothing in the room or is not visible, and if possible, it can be registered separately for floors, walls, ceilings, and the like. If you add furniture and small items one by one and register with the names of the furniture and small items, the newly added part is associated with the registered part name while distinguishing it from the previous environment. It becomes easy.

  In this embodiment, the microphone 156a is also incorporated in the gripping hand 154c, and the voice information signal from the microphone 156a is recognized by the voice recognition unit 156b in comparison with the words registered in the voice recognition unit in advance. It is configured to be able to. A signal pattern of recognizable words is registered in advance before product shipment. For example, a name representative of a home, a word related to the front / rear / up / down / left / right direction, a word related to the robot motion control such as grasping, releasing, pulling, raising, pushing, etc. with the gripping hand 154c. Is registered so that it can be recognized at least. Thus, for example, when the master utters “take a blue cup on the table”, this voice signal is input from the microphone 156a, so that at least the voice recognition unit 156b performs “table” “no” “ If the words “up”, “no”, “cup”, “take” = “take” can be recognized, for example, “object” = “table” “no” “up” as command input to the autonomous control chip 156d. “No” “Copp”, “Operation command” = “Take” = “Grip” and “Move to Master” command can be output. Here, in the case of the command tone, the word “take” is recognized in consideration of a certain ending change because there are “take”, “please take”, and “take off” in addition to “take”. Register so that all words can be recognized, and if any of them is entered, replace it with the word "take" so that it can be processed It is good to leave.

  Here, first, the command to the autonomous control chip 156d is simply composed of “target object” + “type of operation”. The types of movement are grabbing and raising. In response to the command, “grabbing” means “gripping the gripping hand 154c”, and “raising” means “the gripping hand 154c is moved upward from the current position. By defining the correspondence with the content of how to control the operation of the robot and the like, it is possible to prepare an operation program corresponding to the word prepared in advance. If the action command “take” is defined as “gripping” and “moving the object being held in front of the master”, the command “take” “ It is also possible to continuously perform the operation of “move to”. Further, the command to the autonomous control chip 156d is composed of “target object” + “motion type” + “motion description 1” + “motion description 2”, and “motion description 1” and “motion “Description 2 of” is an optional operand. For example, “Description of operation 1” = “To right”, “Description of operation” for the type of operation “Press” = “Move the hand” If it is possible to instruct the operation method of “2” = “slow”, it is possible to control the gripping hand 154c to move slowly to the right from the current state. Therefore, when the autonomous control chip 156d receives a command, a program is prepared so that the arm mechanism 156f or the like is operated to specifically execute the command. When taking a blue cup on the table, first, the object is searched. When there are a plurality of words of “no”, the last word = “cop” is an object, so here, the data of the cup is read out from the environment CAD data for searching and the image information processing apparatus 100 reads the data. Search for position and orientation. If there are multiple search results, it is determined whether it is on the table (search the table to determine whether it is above the coordinate value of the top surface of the table and the coordinate position of the cup) or blue one (This is to identify the position and orientation of the object by checking the color of the searched CAD data and selecting a blue cup as an object that is not blue). Here, the table may be searched first to find what is on it, or a blue cup is specified in advance for the CAD data for search, and both the blue color and the shape of the cup are matched. You may make it search.

  Next, the current position of the gripping hand 154c is obtained based on the CAD data search or the camera position and posture information, and the movement locus of the arm mechanism 156f up to that point and the gripping position from the target cup position and posture information. The final positioning position and posture of the hand 154c are determined, and the autonomous control chip 156d breaks up to the operation command of each joint by the autonomous control chip 156d so that the gripping hand moves to the target position and posture along the trajectory. And output. At this time, the gripping hand 154c is accessed in an open state.

  Next, when the gripping hand 154c is positioned to the target position and posture, that is, the position where the cup 154e can be gripped, the cup 154e is gripped by the gripping hand 154c and slightly lifted to move. After that, the image information processing apparatus 100 is made to search for the current position of the master, and control is performed so as to move to the position while holding the cup 154e. Thus, the operation control for the master command “take a blue cup on the table!” Is completed, and the autonomous control chip 156d responds to the speech recognition / synthesis 156b that the operation has been completed normally. The speech recognition / synthesis 156b is configured to select an appropriate response content prepared in advance, and respond, for example, “Yes. As a result, the words that have been prepared are synthesized by voice, and “Yes. On the other hand, when it is determined that the command cannot be executed during the process, the response from the autonomous control chip 156d is “difficult to execute” and the speech recognition / synthesis unit 156b receives the response from the speaker 156c. "It is difficult." If the cause that the autonomous control chip 156d cannot execute is known and the words are also easy, the reason why it is difficult to execute can be pronounced so that the master can be understood.

  In this embodiment, the command from the master can be input by voice, and the master can confirm the execution result of the command by voice. Therefore, the autonomous robot can be easily operated by the command by words. it can. In the case of a car, if you say where you want to go, it will be an autonomously controlled car that responds with "Where have you arrived?" In addition, if this voice instruction system is applied to a robot that assists a person who needs care, a command can be given by voice instruction and a response can be confirmed by voice. A person who needs care can easily operate the robot alone.

  In this embodiment, the position and orientation information of the arm tip can be obtained by photographing with the camera at the tip and processing by the image information processing apparatus 100. For example, in the case of the six-axis arm mechanism 156f, By omitting angle sensors such as encoders for each joint, and calculating the joint angles for each of the six axes from the position and orientation of the tip obtained by the image information processing apparatus 100, and performing servo control for each axis It can be reasonable. In this case, first, the arm mechanism 156f is caused to perform a somewhat wasteful operation to generate the environmental CAD data, or the arm mechanism 156f is provided while providing a movable end sensor of each joint and referring to the degree of the sensor information. You may make it perform control of. In addition, this embodiment is a wall-hanging type arm mechanism 156f, and this arm mechanism 156f moves along a moving mechanism that travels on the floor surface or a rail that is previously installed in each room or between a plurality of rooms. The moving range may be configured to be wider as the vehicle is mounted on a trolley that can be installed. In addition, the main body of the arm mechanism 156f that operates in the room is made of plastic and each joint mechanism is configured to slide under an overload, so that it is possible to safely cope with a person hitting. It is good to leave.

  Embodiments of the 81st means to the 84th means will be described with reference to FIGS. A wide-range, high-resolution imaging device can be realized by correcting the distortion of image information (video) generated by combining a high-resolution CCD sensor and a fisheye lens. Since this CCD sensor is difficult to use, this embodiment proposes a means for realizing a high resolution camera (imaging device) with a wide field of view relatively easily.

  The basic structure will be described with reference to FIG. The sensor unit is configured by combining a plurality of sets of sensors configured by combining a CCD line sensor 157c and a lens 157d capable of obtaining a relatively high resolution. In this embodiment, a set of sensors is configured to obtain a 90 ° field of view 157m from directly below to the horizontal. If this sensor unit is configured to share a range from directly below to horizontal with a plurality of sensors, the field of view of one set of sensors can be further narrowed, so that the resolution can be increased. The CCD line sensor 157c may be mounted with illumination means (for example, LED) as a set. In order to enable stereoscopic viewing, two sets of line sensors 157c are installed at predetermined intervals in the horizontal direction. The image information signal from the CCD line sensor 157c is input to the image processing LSI 157i, a video signal is generated based on a command from the microcomputer 157h, and can be transmitted and output via the small wireless unit 157j. . A command to the microcomputer 157h is also input via the wireless unit 157j.

  The line sensor 157c and the lens 157d are supported so as to be rotated by the motor unit 157a together with the microcomputer 157h, the image processing LSI 157i, the wireless unit 157j, and the like. The rotating portion is supported by the bearing 157b so as to be stably rotated. The motor unit 157a receives power from a power cord 157k connected to an external power source. The rotation sensor 157f that detects the rotation of the motor unit 157a is attached to the rotating line sensor 157c, and the rotation information output from the rotation sensor 157f is input to the image processing LSI 157i and used as a synchronization signal. The rotation sensor 157f is configured to convert the light and darkness of the slit provided on the fixed side into an electric signal and input the pulse signal as in the optical encoder. Since the rotating position can be detected based on the pulse signal output from the rotation sensor 157f, an image information signal is output from the line sensor 157c in accordance with the rotation signal, and the image information is stored in the image memory in the image processing LSI 157i. By writing, 360 ° image information can be recorded on the memory after one rotation. If the image information signal is overwritten in the memory by rotating the rotating portion at a high speed, moving image information can be generated. As described above, when the image information is generated by the pulse signal of the rotation sensor 157f, normal image information is always obtained even if the rotation speed of the motor unit 157a slightly varies or becomes older and the rotation speed changes. It becomes possible to write to the memory.

  In addition, the rotating portion is provided with a rotating power generation power source 157g that generates electric power by rotating, and a part of the energy rotated by the motor unit 157a is converted into electric power to supply the power of the components attached to the rotating portion To do. Any means may be used as long as the energy required in the rotating part can be generated in the rotating part by a solar battery or microwave instead of the rotating power generation power source 157g.

  With this configuration, it is possible to realize a long-life, wide-field, high-resolution imaging device without using a communication slip ring or a power supply line slip ring. The parts that are highly likely to be worn and damaged due to operation are only the rotating motor part 157a and the bearing 157b, and the other parts are rotating but basically not the driving part. It can be.

  In this embodiment, it is assumed that the camera is suspended and the lower part is photographed. However, if the camera is turned upside down, the camera can be used as a lower part and photographed upward. Further, if the range of the visual field in the horizontal direction is further widened, a wider field of view can be covered. In addition, if it is not necessary to take an image directly below or directly above, the line sensor 157c and the lens 157d may be mounted so as to be inclined so that the range of the field of view is appropriate. An imaging device that is upside down is suitable as an imaging device that is installed on the roof of an automobile or in a passenger seat next to a driver's seat. Moreover, the hanging type imaging device that images the lower side is suitable as a camera that is installed in the lower part of an airship or attached to a streetlight to monitor the road from above.

  In addition, a diagnostic sensor 157e is installed in the rotating part, and a diagnostic signal output from the diagnostic sensor 157e is input to the microcomputer 157h. It is preferable to configure so that a signal can be wirelessly transmitted so that an administrator can be immediately notified when an imaging device used as a surveillance camera fails or is about to fail. The diagnostic sensor 157e may be configured to diagnose the motor unit 157a or the bearing 157b whose life is concerned at the rotating portion, but the diagnostic sensor can confirm that the image information signal is normally output. You may comprise so that it may provide.

  FIG. 73 shows the concept of the image information written in the image memory in the image processing LSI 157i. The horizontal axis indicates 0 ° to 360 °, which is a reference position in the horizontal direction (rotation direction), and the vertical axis indicates a state in which a range from right below to horizontal is read and written. Since this image memory stores high-resolution image information, it may be inconvenient to output all information as an image information signal as it is depending on the capacity of the communication line. From the receiving side, it is possible to instruct the microcomputer 157h in what form the image information signal should be output. Normally, it is configured so that data is thinned out from the entire image information illustrated in FIG. 73 to a suitable resolution and image information of the entire imaging range is output, and a location is partially specified to the microcomputer 157h. When a command for outputting the image information signal as an enlarged (zoomed-up) image information signal is given, the high-resolution image information signal of the corresponding portion may be output at the same resolution without being thinned out. For example, when the area 158b in FIG. 73 is designated, only the image information within the area 158b is output as a high resolution image information signal. In this way, it is possible to extract only the necessary part in detail. Of course, if the combining process is appropriately performed, the rotated CCD line sensor 157c can be a two-dimensional CCD sensor.

  FIG. 74 is a visual information processing apparatus constructed by further developing the embodiment described with reference to FIGS. 72 and 73, and an embodiment in which the image information processing apparatus 100 and the storage device 200 are mounted on the rotating portion. It is. The image information processing apparatus 100 is configured to input image information output from the image processing LSI 157 i, generate CAD data, and store the CAD data in the storage device 200.

  By moving this visual information processing apparatus mounted on a car or the like, the line sensor 157c and the lens 157d are rotated to capture a wide range of environments with high resolution, and image information is generated. Environmental CAD data can be generated and stored in the storage device 100. The signal line 159a from the microcomputer 157h to the image information processing apparatus 100 is a process for generating CAD data executed by the image information processing apparatus 100, a search process for CAD data, a matching process, and a process for tracking a search target in the CAD data. And a signal line for instructing from the microcomputer 157h a process for outputting the result wirelessly.

  In this visual information processing device, the image information processing device 100 and the storage device 200 do not necessarily have to be mounted on the rotating portion, but are installed on the stationary portion, and receive an image information signal transmitted from the wireless unit 157j to receive an image. You may comprise so that it may input into the information processing apparatus 100. FIG.

  In addition, the imaging apparatus described with reference to FIG. 72 is combined with the visual information processing apparatus incorporating the image information processing apparatus 100 and the storage device 200 as described above, and a plurality of pieces of image information are comprehensively processed to convert the environment into CAD data. You may comprise so that it may become. In this configuration, the distance between the line sensors can be set wider than when the line sensor configured for stereo vision is rotated, so that it is possible to improve the 3D measurement accuracy far away.

  An embodiment of the 85th means and the 86th means will be described with reference to FIG. In this embodiment, environmental CAD data generated based on image information obtained by photographing a plurality of automobiles equipped with visual information processing devices (terminals) arbitrarily traveling around a nationwide road network is centrally processed. In the central information processing center, an environmental CAD data received from each terminal is integrated to generate nationwide environmental CAD data, and distributed to the terminal in response to a request from each terminal. is there.

  A plurality of vehicles equipped with terminals (visual information processing devices) 160a to 160c equipped with an imaging device and an image information processing device were obtained by imaging each of the terminals 160a to 160c while arbitrarily driving around towns throughout the country. Based on the image information, CAD data is generated and wirelessly transmitted to the central information processing center 160e, and the central information processing center 160e generates CAD data around the national road network by integrating the received CAD data. To do. Each terminal 160a, 160b, 160c also transmits current information (mainly position and orientation information) to the central information processing center 160e. Then, the central information processing center 160e distributes the environment CAD data and other terminal information generated by integration to the terminals 160a to 160c in response to requests from the respective terminals. Here, if the visual information processing apparatus described above is used as the terminals 160a to 160c, CAD data can be easily generated and integrated.

  Service fees for this system are collected by adding each terminal (user) 160a-160c as a monthly fee or annual fee, or by adding fees based on the usage time or amount of data used for integrated CAD data. The method can be adopted.

  In addition, this system allows a user who does not have a terminal of a visual information processing apparatus to access the central information processing center 160e from a terminal 160d such as a personal computer via the Internet or the like and charge integrated CAD data for a fee. It can be configured to be used. Further, CAD data integrated using a recording medium such as a CD can be supplied to the user.

  An embodiment of the 87th means and the 88th means will be described with reference to FIG. This embodiment is an environment monitoring service system.

  A monitoring camera 161b is installed on a streetlight 161a or the like. The monitoring camera 161b is desirably an imaging device that can capture the wide range described with reference to FIG. 72 with high resolution. The monitoring camera 161b is configured to transmit image information generated by photographing to the monitoring center 161g by wireless communication. The wireless communication means is configured to use a mobile phone line, so that image information captured by the monitoring camera 161b can be sent to the monitoring center 161g via the telephone company 161e and the network line 161f.

  The user who requests the monitoring service is configured to apply for the monitoring service (monitoring service ON) to the monitoring center 161g by, for example, PHS or a mobile phone. The monitoring center 161g obtains an outline of the position of the user by referring to the communication path of the mobile phone or the PHS, identifies the user using the video based on the image information from the monitoring camera 161b, and shoots the environment of the user. It is a service that monitors safety. If the user moves, the imaging area is moved following the movement. The tracking of the user is configured to perform automatic tracking by processing the image information of the user captured on the screen (image information). In the case where the user moves across the imaging areas of the plurality of monitoring cameras 161b, it is possible to continuously monitor even when a state of overlapping monitoring occurs in the image information of both the monitoring cameras 161b. .

  With this configuration, when the user is likely to be harmed by the suspicious person 161d, the monitoring center 161g notifies the police or warns the suspicious person 161d from the speaker attached to the monitoring camera 161b. , So that harm can be prevented in advance. In the case of being injured, the image information at that time is recorded in the monitoring center 161g, so that it can be referred to when identifying the suspicious person who has done the harm.

  If the user wears a wearable surveillance camera as described with reference to FIG. 65 and is configured to receive the surveillance service by sending image information obtained by photographing with the surveillance camera to the surveillance center 161g, A monitoring service can be received even in a place where a surveillance camera is not installed.

  If this monitoring service system is configured to record the time-series change information of the three-dimensional CAD data generated by using the visual information processing apparatus described above, the situation of the site can be more accurately monitored or suspicious. It is possible to provide information for identifying a person.

  The service fee can be collected as a telephone call volume linked to the service time or as a membership fee system.

  As described above, according to the present invention, CAD data is generated based on image information obtained by moving an image pickup apparatus, and highly accurate CAD data is generated for the same target portion as the existing CAD data. Since the existing CAD data is updated with the new CAD data, it is not necessary for the person to input and set the corresponding feature points in the plurality of photos at first. It is possible to generate accurate CAD data without mapping, to generate CAD data with a certain degree of accuracy at first, and to be able to use it for control of robots from the beginning, or to shoot with multiple imaging devices CAD data generated based on the obtained image information can be easily integrated, and further, as-built CAD data is generated. Applying it to a visual guidance control device of an automatic machine such as a robot, a navigation system of an automobile, etc., or applying it to an autonomous control robot, etc. Etc. can be easily performed. In other words, as-built 3D-CAD data can be automatically generated in real time, and in combination with a CAD data search function, the visual information can be applied as a visual guidance control device for various automatic machines such as robots, and as a navigation system. A processing apparatus and its application system can be obtained.

It is a block diagram which shows the basic composition of the 1st means and the 2nd means in the visual information processing apparatus of this invention. It is a block diagram which shows the basic composition of the 3rd means and the 4th means in the visual information processing apparatus of this invention. It is a flowchart which shows the basic flow of a process of the image information processing apparatus which implement | achieves a 3rd means and a 4th means. It is drawing which illustrates the content of the process (1) shown in FIG. It is drawing which illustrates the content of the process (2) shown in FIG. It is drawing which illustrates the content of the process (3) shown in FIG. It is drawing which illustrates the content of the process (4) shown in FIG. It is drawing which illustrates the content of the process (5) shown in FIG. It is a figure which shows the concept of utilization of the floating line segment produced | generated as CAD data. It is a figure which shows the concept of utilization of the floating line segment produced | generated as CAD data of a cylinder or a sphere. It is a figure which shows an example of the definition as CAD data of a special curved surface. It is a conceptual diagram which shows embodiment in the case of updating CAD data in the 1st means, 2nd means, and 3rd means of this invention. It is a conceptual diagram which shows embodiment of the 5th means of this invention. It is a conceptual diagram which extracts an outline using the color information in the 8th Embodiment of this invention. It is a conceptual diagram which shows embodiment of the 13th means-18th means of this invention. It is a conceptual diagram which shows embodiment of the 13th means-18th means of this invention. It is a flowchart of the mirror surface discrimination | determination process in this invention. It is a figure which shows the content which can be discriminate | determined by the mirror surface discrimination | determination process shown in FIG. It is the image information explaining embodiment of the 19th means-the 21st means of the present invention. It is a block diagram which shows embodiment of the 22nd means-26th means of this invention. It is a block diagram which shows the 27th means-29th Embodiment of this invention. It is a block diagram which shows 27th means-29th other embodiment of this invention. It is a block diagram which shows other embodiment of the 27th means-29th means of this invention. It is a block diagram which shows other embodiment of the 27th means-29th means of this invention. It is a flowchart which shows embodiment of a process of the visual information processing apparatus in the case of implementing the 27th means-29th means of this invention as a visual information processing apparatus only for robot control. It is a conceptual diagram explaining embodiment of the 30th means, 33rd means, 34th means, and 39th means of this invention. It is a conceptual diagram explaining embodiment of the 30th means, 33rd means, 34th means, and 39th means of this invention. It is a conceptual diagram explaining embodiment of the 31st means and 32nd means of this invention. It is a conceptual diagram explaining embodiment of the 35th means-38th means of this invention. It is a conceptual diagram explaining embodiment of the 35th means-38th means of this invention. It is a conceptual diagram explaining embodiment of the 35th means-38th means of this invention. It is a conceptual diagram explaining embodiment of the 35th means-38th means of this invention. It is a conceptual diagram explaining embodiment of the 40th means and 41st means of this invention. It is a block diagram of the visual information processing apparatus which is the embodiment of the forty-second means of the present invention. It is a conceptual diagram explaining embodiment of the 43rd means-50th means of this invention. It is a display screen in the 43rd means-50th embodiment of this invention. It is a block diagram which shows the basic composition of the visual information processing system which is embodiment of the 51st means of this invention. It is a conceptual diagram of CAD data for movement information acquisition in the visual information processing system shown in FIG. It is a conceptual diagram of CAD data for movement information acquisition in the visual information processing system shown in FIG. It is a conceptual diagram of CAD data for movement information acquisition in the visual information processing system shown in FIG. It is a conceptual diagram of CAD data for movement information acquisition in the visual information processing system shown in FIG. It is a schematic diagram of the mobile work vehicle which is embodiment of the 52nd means of this invention. It is a conceptual diagram explaining embodiment of the 53rd means and 54th means of this invention. It is a conceptual diagram explaining embodiment of the 55th means-57th means of this invention. It is a block diagram of the visual information processing apparatus which is the embodiment of the 58th means of the present invention. It is a flowchart of the main process which the visual guidance chip | tip in embodiment shown in FIG. 45 performs. It is a flowchart of the main process which the visual guidance chip | tip in embodiment shown in FIG. 45 performs. 48 is a detailed flowchart of 3D coordinate calculation and CAD data generation processing in processing (2) shown in FIG. 47. 49 is a detailed flowchart of additional generation processing in the 3D-CAD generation processing of processing (4) and processing (9) shown in FIG. 48. 49 is a detailed flowchart of a point data update process in the 3D-CAD generation process of the process (4) and the process (9) shown in FIG. 48. 49 is a detailed flowchart of line segment data update and addition processing in the 3D-CAD generation processing of processing (4) and processing (9) shown in FIG. 48. 49 is a detailed flowchart of surface data update and addition processing in the 3D-CAD generation processing of processing (4) and processing (9) shown in FIG. 48. FIG. 53 is a conceptual diagram of data update in the 3D-CAD data generation process shown in FIGS. 49 to 52. FIG. 53 is a diagram depicting the concept of generation and update of new data in the 3D-CAD data generation process shown in FIGS. 49 to 52 by applying it to the concept of generation and update of general graphics. It is a schematic diagram of the mobile work vehicle which is an embodiment of the 59th means of the present invention. It is a schematic diagram of the visual information processing system which is an embodiment of the 60th means of the present invention. It is a schematic diagram of the manipulator which is embodiment of the 61st means and 62nd means of this invention. It is a schematic diagram of underwater ROV which is an embodiment of the 62nd means of the present invention. It is a schematic diagram of the automatic cleaner which is embodiment of the 62nd means of this invention. It is a schematic diagram of the character reader which is embodiment of the 63rd means of this invention. It is a schematic diagram of the three-dimensional measuring apparatus which is the 64th means-the 66th means of this invention. It is a conceptual diagram explaining embodiment of the 67th means of the present invention. It is a conceptual diagram explaining the navigation apparatus which is embodiment of the 68th means of this invention. It is a display screen of the LCD monitor in the car navigation apparatus which is the 69th embodiment of the present invention. It is a schematic diagram explaining embodiment of the navigation apparatus with which a pedestrian wears. It is a block diagram of the picking system which is an embodiment of the 70th means of the present invention. It is a schematic diagram of the inspection apparatus which is embodiment of the 71st means and 72nd means of this invention. It is a block diagram of a home robot which is an embodiment of the 73rd means to the 79th means of the present invention. It is a schematic diagram of the home robot which is embodiment of the 73rd means-the 79th means of this invention. FIG. 69 is a flowchart of an autonomous control process executed by a microprocessor of the pet robot shown in FIG. 68. FIG. 70 is a block diagram of the home robot shown in FIG. 69. It is a schematic diagram of the imaging device which is embodiment of the 80th means-84th means of this invention. FIG. 73 is a conceptual diagram of image information written in an image memory of the imaging apparatus shown in FIG. 72. It is a schematic diagram of the visual information processing apparatus which is an embodiment of the 80th to 84th means of the present invention. It is a schematic diagram of the service system which is embodiment of the 85th means and the 86th means of this invention. It is a schematic diagram of the environmental monitoring service system which is an embodiment of the 87th means and 88th means of the present invention.

Explanation of symbols

  10, 10a, 10b, 20 ... imaging device (camera), 30 ... acceleration sensor, 40 ... gyro sensor, 100, 120a, 120b, 120c ... image information processing device, 200 ... storage device.

Claims (4)

In autonomous control robots,
An autonomous control robot characterized in that the safety of the operation of the robot can be monitored by an independent system different from the autonomous control system. In the control device for autonomous control robots,
A control apparatus for an autonomous control type robot, wherein the safety of operation of the robot can be monitored by an independent system different from the autonomous control system. In autonomous control robots,
An autonomous control robot characterized in that emergency response control of a robot can be performed by an independent system different from the autonomous control system when emergency response is required during safety monitoring of the robot. In the control device for autonomous control robots,
Control of an autonomously controlled robot characterized in that emergency response control of the robot can be performed by an independent system different from the autonomous control system when emergency response is required during safety monitoring of the robot apparatus.

Images