JP2011151636A - Compound eye camera and camera application equipment - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a compound eye camera which generates an image having an expanded viewing angle, and measures distance of an object with a viewing angle nearly equal to that of a camera image from the camera image, and to provide camera application equipment which attains a new functions, is low-cost and has high performance. <P>SOLUTION: A wide image with a visual field of 2α is generated and the distance is measured by arranging two cameras a, b with the same horizontal view angle α in such a manner that the optical axes of the cameras a, b cross each other at the same angle α as a horizontal view angle α on a plane including the lens centers of the two cameras a, b, and by compositing images. Furthermore, in the case of a four-eye camera which is similarly arranged, a picture with a visual field of 2βis generated also in the vertical direction, and a distance measurement range is also similarly expanded. In addition, an image information data amount of transmission and recording is reduced, and at the same time storage of things left behind and place finding game, etc. are played by storing a background image to record change of background by a moving object/person extraction part, body type information of the person is left as growth record from motion start part information, and a function such as grading of dance, etc. is provided from comparison with motion information of a teacher. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a compound-eye camera and a camera application device, and more particularly to a compound-eye camera, a game machine, an automobile, and an apparatus capable of synthesizing an image with an enlarged viewing angle from a plurality of camera images and measuring a subject distance at the same viewing angle. The present invention relates to a camera application device that realizes a new function in WEB or the like.

  In recent years, in many fields such as game machines, home appliances, crime prevention, automobiles, and robots, practical application of image recognition technology and distance measurement technology in combination has become active. Unlike general high-quality digital cameras, image recognition technology uses a relatively small number of inexpensive cameras with a relatively small number of pixels. In the future, functions and performance will become more attractive, including the provision of wide screens. Therefore, a camera with a wide viewing angle is expected.

  In order to realize a camera with a wide viewing angle, there are a method using a wide-angle lens and a method for realizing by combining a captured image of a compound eye camera. However, wide-angle lenses are expensive and have a problem that requires distortion correction processing to cause barrel distortion. Also, since a wide-field image is acquired with an image sensor for a standard lens, there is a problem that resolution deteriorates. is there.

  As a method using a compound eye camera, many binocular panoramas have been proposed as a method for acquiring an image having a double angle of view. For example, in the proposal of “Panorama Camera Arrangement Method and its Construction Method” in Patent Document 1, a method has been devised in which imaging lenses are arranged so as to overlap in the vertical direction, the center of the lens in the horizontal direction is aligned, and there is little parallax. However, even with this method, parallax remains, and there remains a major problem of vertical field angle shift in a close image in the vertical direction and a problem of vertical parallax.

  On the other hand, in the distance measurement technology, millimeter wave radar and ultrasonic sonar are mainly used for in-vehicle applications, and infrared rays are mainly used for other applications. However, radio waves, sound waves, and infrared transmitters are required, and distance measurement is performed using the time difference of reflection. Therefore, when measuring multiple points at the same time, an expensive sensor is required, and a wide angle range is required. No distance measurement has been realized. In addition, there is a problem in linked processing such as difficulty in directly relating to the image recognition result.

  Therefore, an approach in which image recognition and distance measurement are performed simultaneously from camera image information is being studied. A representative example is the stereo camera approach that is being studied for in-vehicle applications. However, the conventional stereo camera has a problem that a composite image having a viewing angle larger than the horizontal angle of view of the lens cannot be obtained and a range in which the distance can be calculated is less than the horizontal angle of view of the lens.

  Next, looking at applications using camera images and distance information, for example, there are those using motion recognition in motion control of game machines and home appliances. However, most of the approaches using distance information are, for example, those using a TOF (Time Of Flight) type infrared sensor or the like, or using a remote control using an image and an acceleration sensor. However, there is a problem that the calculation burden of the game machine is heavy, such as processing camera information on the game machine body. Also, security cameras have almost all functions such as simply viewing or recording video on a display, and there are very few functions that are truly effective using image recognition.

JP 2002-369060 A

  The present invention has been made to solve such problems, and has realized a camera with a wide viewing angle that is twice or more the angle of view of a lens, and at the same time, distance measurement from image information with respect to a wide viewing angle. An object of the present invention is to provide a compound eye camera that can be used, and to realize a new function in a camera application device, particularly in a camera used at a fixed point.

  In the first aspect of the present invention, two optical cameras a and b having the same horizontal angle of view α (or vertical angle of view β) are provided, and the optical axis a2 of the camera a on a plane including the lens centers of the cameras a and b. And the optical axis b2 of the camera b are arranged so as to intersect at the same angle α (or β) as the horizontal angle of view α (or vertical angle of view β). In the following, a straight line passing through a point where two optical axes intersect and having the same angle as the two optical axes (that is, α / 2 or β / 2) on the same plane will be expressed as an average optical axis Xab. To do.

In the invention of claim 2, among the four cameras a, b, c, d having the same horizontal angle of view α, both of the cameras a, b or cameras c, d are the compound eye cameras of claim 1, The optical axis of the camera a and the optical axis of the camera c are arranged in parallel on a plane including the four lens centers, and the optical axis of the camera b and the optical axis of the camera d are arranged in parallel.

According to a third aspect of the present invention, there are provided four identical cameras a, b, e, and f having a horizontal angle of view α and a vertical angle of view β, and both the cameras a and b and the cameras e and f are described in the first aspect. Are arranged so as to intersect at an angle β on a plane formed by the average optical axis Xab of the cameras a and b and the average optical axis Xef of the cameras e and f.

  Further, as in the invention of claim 4, at least one compound eye camera of claims 1 to 3 may be provided, and these compound eye cameras (/ and single camera) may be combined.

  Further, as in the invention of claim 5, the compound-eye camera of claims 1 to 4 may be provided, and an image composition unit may be provided.

  Further, as in the invention of claim 6, the compound eye camera of claims 1 to 5 may be provided, and a distance calculation unit for calculating the distance of the overlapping region of view angles may be provided.

  According to a seventh aspect of the invention, a compound eye camera, a subject three-dimensional coordinate calculation unit, a virtual camera information input unit, and a virtual imaging information distance calculation unit are provided, and subject three-dimensional coordinates are obtained from image outputs of a plurality of cameras of the compound eye camera. The calculation unit calculates the three-dimensional coordinate value of the subject, calculates the virtual imaging information distance calculation unit from the three-dimensional coordinate value, the virtual lens center coordinate value and the virtual optical axis vector value from the virtual camera information input unit, A composite image equivalent to a pinhole camera at the center of the lens is obtained.

  Further, as in the eighth aspect of the invention, the compound-eye camera according to the seventh aspect includes an optical axis convergence point calculation unit, and from the virtual camera information input unit, two virtual optical axis vectors of the virtual 3D camera and a three-dimensional object. The three-dimensional coordinate value of the subject calculated by the coordinate calculation unit is input to the optical axis convergence point calculation unit, and two recalculated optical axis vectors are calculated by recalculating the optical axis convergence point. The calculation unit calculates virtual imaging information corresponding to the two lenses, and outputs 2-channel composite image data and distance data.

  Next, means for solving the problems related to the camera application apparatus according to claims 9 to 17 of the present invention will be described. The invention according to claim 9 comprises a camera, a background image storage unit, a moving object person incidental extraction unit, and a background image calculation control unit, and includes imaging information from the camera and the present moving object person from the moving object person extraction unit. When the object information and the current background image are input to the background image calculation control unit and it can be determined from the moving object person accompanying information that the accompanying object has left the moving object or / and the person, the accompanying object is removed. The determination part is overwritten on the current background image, and the background image storage unit is updated as the next background image.

  The invention according to claim 10 relates to the camera application device according to claim 9, comprising a background change storage unit, a timer, a background change confirmation control unit, and an initial background image storage unit, and simultaneously updating the background image storage unit. The change image information different from the background of the time is stored in the background change storage unit together with the time series information, and the background image at the specified time is changed from the image of the initial background image storage unit and the background change by the instruction of the background change confirmation control unit. It is characterized in that a background image can be regenerated and confirmed at any time from an image in a storage unit.

  Further, as in the invention of claim 11, the camera application device of claim 9 is provided with a host device or / and an in-camera image storage device, and the host device or / and the in-camera image storage device in advance. The initial information of the background image storage unit is transmitted, and thereafter, the update information is transmitted to the host every time the background image storage unit is updated, and the information of the moving object / person extraction unit is transmitted as needed to The transmission information amount of camera information and the storage device capacity are reduced.

  Further, as in the twelfth aspect of the present invention, the camera application device according to the ninth aspect includes a motion extraction unit that extracts motion information from the host device and the moving object human accompanying extraction unit, and only the motion information is transmitted through a communication line. May be transmitted to the host device or the like to reduce the amount of data communication to the host device.

  Further, as in the invention of the thirteenth aspect, the data transmission to the host device of the eleventh or twelfth aspect may be realized at the same time.

  Further, as in the invention of claim 14, the camera application device according to claim 13 is provided with a motion scoring unit for scoring by comparing the motion information sent to the image display device and the host side with the teacher motion information on the host side. The teacher image and the human camera image may be simultaneously displayed on the divided display area on the image display device, and the scoring may be performed sequentially.

  Further, as in the fifteenth aspect of the present invention, the camera application device according to the ninth aspect of the present invention includes an edge extraction unit and a body shape information calculation unit, and the edge extraction unit uses the human image information obtained from the moving object person accompaniment extraction unit. A configuration may be adopted in which edge information is extracted, edge information in all directions obtained by rotation of a person is collected, and feed body shape information is calculated in the camera or to the host device.

  Further, as in the invention of claim 16, the camera application device of claim 15 is provided with a display device, a body shape information storage unit, a timer, and a body shape display control unit. The date and time information may be recorded in the body shape information storage unit so that temporal changes can be displayed.

  Further, as in the invention of the seventeenth aspect, the camera application apparatus according to the ninth to sixteenth aspects may include the compound eye camera according to the first to eighth aspects.

  The present invention can be realized not only as an apparatus or a system but also as a method. Of course, a part of the invention can be configured as software. Of course, software products used to cause a computer to execute such software are also included in the technical scope of the present invention.

  The above-described aspects and other aspects of the present invention are described in the claims, and will be described in detail below using examples and the like.

  As in the first aspect of the invention, by arranging the optical axes of the two cameras so as to intersect at an angle α on a plane including the lens centers of the two cameras having a horizontal angle of view α, a camera image at an angle 2α is obtained. Information can be acquired, and if it is processed by the image composition unit of claim 5, one wide-angle image can be generated.

  Further, as in the invention of claim 2, the optical axes of the four cameras having the horizontal angle of view α intersect in the same plane and the two intersect at an angle α. By locating the remaining two optical axes in parallel with the two optical axes, the distance can be measured by processing the distance calculation unit of claim 6 using the triangulation principle at an angle of 2α.

  Further, if these lenses are lenses having a vertical angle of view β as in the invention of claim 3, if the same measures are taken in the vertical direction, the angle at the angle of 2α in the horizontal direction and 2β in the vertical direction is obtained. If processing is performed by the image composition unit and the distance calculation unit simultaneously with the acquisition of the camera image, a two-dimensionally wide image can be generated and the distance can be measured. Note that if the multiples of the angle of view α and β are in the range of less than 180 degrees, it is possible to enlarge the angle of image acquisition and distance measurement to 3 times and 4 times.

  If the subject three-dimensional information calculation unit and the optical axis convergence point calculation unit are provided as in the seventh aspect of the invention, 2-channel 3D image information can be obtained by calculation using information obtained from these compound-eye cameras. It becomes possible.

  Next, according to the invention relating to the camera application device of claim 9 to claim 17, when an object accompanying the moving object (such as a car or a person) appears in the stored initial background image, the difference from the background image is determined. When the change determination unit determines that the object has moved away from the moving object, if there is a change in the background, it can be determined that something other than the moving object has been placed in the background. If a history of changes is stored, figurines can be checked back in time, leading to lost object searches, new games, and the like. In addition, it is possible to reduce the amount of data by storing only this changed part and transferring it to the host at any time.

  As described above, in the case of the compound eye camera according to any one of claims 1 to 4, the angle of view more than twice the horizontal angle of view of the camera, the angle of view of the vertical angle of view more than twice, and a wide viewing angle. It is possible to realize a camera that can acquire the imaging information, and can acquire the distance measurement information with respect to the region having the overlapping angle of view. When used as an image input camera such as a personal computer, an image with a wide viewing angle can be created by performing image composition such as the panoramic composition method in a personal computer, and distance calculation using distance calculation software is possible. There is.

  The compound eye camera according to claim 5 has a built-in image synthesizing unit, so that an image with a wide viewing angle can be created by the camera alone, and the compound eye camera according to claim 6 has a built-in distance calculation unit to measure the distance by the camera alone. There is an effect that can be done.

  In the case of the compound eye camera according to claim 7, since the three-dimensional information of each point of the subject can be obtained by calculation, the image taken by the virtual pinhole camera and the distance from the center of the virtual lens can be obtained simultaneously. For an object with both near and near, it is possible to obtain an image with a wide viewing angle superior to the panorama synthesis technique.

  In the case of the compound eye camera according to claim 8, the determination function by the optical axis converging unit allows the intersection of the optical axes to be matched with the nearest object point, and the two-channel video suitable for 3D stereoscopic video can be taken. There is an effect to become.

  The camera application device according to claim 9 is assumed to be a camera application device installed at a fixed position for a certain period of time, and processes an incidental object such as a moving object such as a car or a person and an object held by a person at any time, There is an effect that the background image can be sequentially updated.

  In the camera application device according to the tenth aspect, there is an effect that it is possible to generate a background screen in which time is specified from information in the initial screen storage unit and the background change storage unit.

  In the case of the camera application device according to the eleventh aspect, by processing the imaging information from the camera into initial background information and subsequent moving object person accompaniment information, processing to the capacity of the in-camera storage device or to the host There is an effect that the communication capacity can be reduced.

  In the case of the camera application device according to the twelfth aspect, since only the motion information is transmitted to the host device, the processing load on the host device such as a game machine whose processing capability is limited by 3D graphics or the like is reduced. In this way, applications such as motion control can be realized.

  In the case of the camera application device according to the thirteenth aspect, there is an effect that the eleventh and twelfth aspects can be realized simultaneously.

  In the case of the camera application device according to the fourteenth aspect, it is possible to realize an application such as a dance scoring game in which scoring is performed while both the teacher and the user are displayed on the screen.

  In the camera application device according to the fifteenth aspect, there is an effect that it is possible to acquire the human body shape information. In the case of the camera-applied device according to claim 16, the body shape information of the person is recorded, the body shape information at the time when it is desired to be confirmed, and the growth graph is viewed by the body shape information such as the growth graph is realized. There is an effect that can be done.

  In the case of the camera application device of claim 17, the effect of claims 9 to 15 can be extended to camera information of a wide viewing angle image by using the compound eye camera of claims 1 to 8 for the camera unit. In particular, it has the effect of connecting to a wide range of applications, such as playing games in the room, wide-screen security cameras, and video conferencing where all participants are shown.

It is a compound eye camera lineblock diagram of the 1st example of the present invention. It is a compound-eye camera field angle overlapping figure of the 1st example of the present invention. It is a compound-eye camera block diagram of the 2nd Example of this invention. It is a compound-eye camera field angle overlap figure of the 2nd example of the present invention. It is a compound-eye camera block diagram of the 3rd Example of this invention. It is a compound-eye camera field angle overlap figure of the 3rd example of the present invention. It is a side view of the compound eye camera structure of the 4th Example of this invention. It is a compound eye camera block diagram of the 5th Example of this invention. It is a compound-eye camera field angle overlap figure of the 5th example of the present invention. It is a compound eye camera block diagram of the 6th Example of this invention. It is a form figure using the personal computer of the compound eye camera of the 6th example of the present invention. It is a compound eye camera block diagram of the 7th Example of this invention. It is a compound eye camera block diagram of the 8th Example of this invention. It is a compound eye camera block diagram of the 9th Example of this invention. It is a compound eye camera block diagram of the 10th Example of this invention. It is a camera application apparatus block diagram of the 11th Example of this invention. It is an accompanying detachment process explanatory drawing of the 11th Example of this invention. It is a camera application apparatus block diagram of the 12th Example of this invention. It is a camera application apparatus block diagram of the 13th Example of this invention. It is a camera application apparatus block diagram of the 14th Example of this invention. It is a camera application apparatus block diagram of the 17th Example of this invention. It is a camera application apparatus block diagram of the 18th Example of this invention.

  The information browsing system according to the embodiment of the present invention will be described below.

  First, an embodiment of the compound eye camera of the present invention will be described.

[First embodiment]
FIG. 1 is a block diagram of a compound eye camera according to a first embodiment of the present invention relating to claim 1. In FIG. 1, the camera is composed of a camera a having a lens center a1 and a camera b having a lens center b1, which are lenses having the same horizontal angle of view α. The optical axes are a2 and b2, the imaging elements are a3 and b3, and the lens center a1. And b1 are arranged so that the optical axis a2 and the optical axis b2 intersect at an angle α, so that the total range of angle of view in the image sensor of the camera a and the camera b is 2α. Is possible. The camera of the present invention may be used for still images or moving images depending on the application.

  FIG. 2 shows the overlapping view angle overlap of the first embodiment of the present invention in which the overlapping portions of the view angles of the two cameras a and b in FIG. FIG. In the conventional arrangement of compound eye cameras, the total range of angles of view shown on the image sensor is less than 2α, whereas the camera arrangement of claim 1 can set the angle range shown on the image sensor to 2α. In addition, the arrangement is such that an overlapping portion in the horizontal direction of an image required for panorama synthesis processing or the like can be guaranteed. Further, in the overlapping portion, distance measurement is possible based on the principle of triangulation.

[Second Embodiment]
FIG. 3 is a block diagram of a compound-eye camera according to a second embodiment of the present invention relating to claim 2. In FIG. 3, the compound-eye camera comprising a and b of claim 1 and the compound-eye camera comprising c and d are parallel to each other on the same plane, and the optical axes of the camera b and camera d are parallel to each other. Then, the cameras a and b are connected to the camera c. , D are arranged inside. This can also be viewed as comprising the compound eye camera of claim 1 comprising c and b and the compound eye camera of claim 1 comprising a and d. FIG. 4 is an overlapping view of compound eye camera angles of view according to a second embodiment of the present invention relating to claim 2.

  As can be seen from FIG. 4, the angle range having the horizontal overlap of the camera angle of view required by the triangulation method is an arrangement extending to 2α if the overlap of the four cameras is combined. Yes. Therefore, the distance measurable range can be 2α.

  Also, the overlap in the center from FIG. 4 indicates that the distance can be calculated by averaging processing or the like because distances can be calculated from six combinations of cameras a, b and a, c and a, d and b, c and b, d and c and d. Accuracy can be increased. This is particularly effective when it is desired to increase the distance calculation accuracy in the central portion.

[Third embodiment]
FIG. 5 is a structural view of a compound eye camera according to a third embodiment of the present invention relating to claim 2. In FIG. 5, the compound-eye camera consisting of a and b and the compound-eye camera consisting of c and d are arranged in a symmetrical positional relationship. FIG. 6 is an overlapping view of compound-eye camera angles of view according to the third embodiment of the present invention relating to claim 2. FIG. 6 shows an arrangement in which the angle range having the horizontal overlap of the field angles of the two cameras is 2α as in FIG. Therefore, the distance measurable range can be 2α. In triangulation, it is known that it is advantageous to increase the distance between lenses when increasing the distance measurement accuracy in a distant place, so it is effective when priority is given to distance accuracy over long distances. It is an example.

[Fourth embodiment]
FIG. 7 is a side view of the compound-eye camera configuration of the fourth embodiment of the present invention relating to claim 3. In FIG. 7, the camera includes four cameras a, b, e, and f having the same horizontal angle of view α and vertical angle of view β, and both the cameras a and b and the cameras e and f are the compound-eye cameras according to claim 1. By arranging Xab and Xef so as to intersect at an angle β on the plane formed by the average optical axis Xab of cameras a and b and the average optical axis Xef of cameras e and f, imaging of cameras a, b, c and d is performed. With respect to the range of the angle of view shown in the element, the total in the horizontal direction is 2α, and the total in the vertical direction can be 2β. Although not shown in the figure, there is also an overlapping portion in the vertical direction and a range in which distance measurement is possible.

[Fifth embodiment]
FIG. 8 is a block diagram of a compound eye camera according to the fifth embodiment of the present invention relating to claim 4. FIG. 9 is an overlapping view of compound eye camera angles of view of the fifth embodiment of the present invention relating to claim 4.

  In FIG. 8, a single-lens camera e is arranged in the center of the configuration of FIG. 5 which is the third embodiment. As can be seen from FIG. 9, the accuracy of distance measurement can be increased by increasing the overlap of the angle of view, and for example, when shooting is possible only with the camera e, such as when there is only one subject, There is an advantage that the power of the other four cameras can be turned off.

[Sixth embodiment]
FIG. 10 is a block diagram of a compound eye camera according to the sixth embodiment of the present invention relating to claims 1 and 4. In FIG. 10, the cameras a, b, and c having the same horizontal angle of view α are configured. The cameras a and b correspond to the compound-eye camera of claim 1 and the single-lens camera c corresponding to claim 4 is added. However, the two cameras b and c also have the structure of the compound eye of claim 1. In this case, the total range of the angle of view shown on the image sensors of the cameras a, b, and c can be 3α. In addition, the area with the addition of the angle of view has two V-shaped areas, and the distance can be measured. This is effective for obtaining wide-angle shooting information from a standard lens with little lens distortion. Similar to the sixth embodiment, the present invention is not limited to the horizontal direction and is not limited to the trinocular camera, and according to claims 1 to 4 of the present invention, the same configuration is realized in the vertical direction and with four or more eyes. Obviously we can do it.

  In addition, since the first to fourth aspects of the present invention are inventions related to the configuration of a compound eye camera, the image synthesis and / or distance calculation can be realized by a configuration using a personal computer. FIG. 11 shows the configuration of the sixth embodiment of the present invention showing a configuration in which imaging information 1102 from the compound eye camera 1101 is input to the personal computer 1103 and the personal computer 1103 outputs the image composition data 1104 with image composition software such as a panorama composition method. It is a form figure using the personal computer of a compound eye camera. The same configuration can be used when distance calculation is performed instead of image synthesis or both are calculated.

[Seventh embodiment]
FIG. 12 is a block diagram of a compound eye camera according to the seventh embodiment of the present invention relating to claim 5. In FIG. 12, imaging information 1202 from the compound-eye camera 1201 is input to the image composition unit 1203, and the image composition unit 1203 performs image composition processing such as panorama composition and outputs image composition data 1204. As a result, wide screen information can be output.

[Eighth embodiment]
FIG. 13 is a structural view of a compound eye camera according to an eighth embodiment of the present invention relating to claim 6. In FIG. 13, imaging information 1302 from the compound-eye camera 1301 is input to the distance calculation unit 1303, and the distance calculation unit 1303 performs distance calculation processing based on the principle of triangulation, for example, and outputs distance calculation data 1304. This makes it possible to output the distance calculation result of the overlapping region of view angles.

[Ninth Embodiment]
FIG. 14 is a structural view of a compound eye camera according to the ninth embodiment of the present invention relating to claim 7. In FIG. 14, the imaging information 1402 from the compound eye camera 1401 is input to the subject three-dimensional coordinate calculation unit 1403, and the subject three-dimensional coordinate value 1404 is calculated and output. The virtual imaging information distance calculation unit 1408 calculates the composite image data 1409 and the distance calculation data 1410 with the lens center coordinate value 1405, the optical axis vector 1406, and the subject three-dimensional coordinate value 1404 input from the virtual camera information input unit 1407 as inputs. And output. This makes it possible to simultaneously obtain a composite image and distance information corresponding to the pinhole camera at the center of the virtual lens.

[Tenth embodiment]
FIG. 15 is a block diagram of a compound eye camera according to the tenth embodiment of the present invention relating to claim 8. In FIG. 15, the imaging information 1502 from the compound-eye camera 1501 is input to the subject three-dimensional coordinate calculation unit 1503, and the subject three-dimensional coordinate value 1504 is calculated and output. The optical axis convergence point calculation unit 1509 receives the automatic on / off switch 1508 input from the virtual camera information input unit 1505, the two optical axis vectors 1507 of the 3D camera set in advance manually, and the subject three-dimensional coordinate value 1504, When the automatic on / off switch 1508 is on, for example, two optical axis vectors 1507 are corrected and calculated so that the point of the subject closest to the subject becomes a convergence point, and two corrected optical axis vectors 1510 are output. The output two lens center coordinate values 1509, the virtual imaging information distance calculation unit 1511, the subject three-dimensional coordinate value 1504, the two corrected optical axis vectors 1510, and the two lens center coordinate values 1506 from the virtual camera information input unit. Input, and calculate as imaging information from two virtual lenses with two corrected optical axes as the line-of-sight direction, and calculate and output 2-channel composite image data 1512 and, if necessary, 2-channel pixel point distance data 1513.

  When the automatic on / off switch 1508 is off, the optical axis convergence point calculation unit 1509 outputs two optical axis vectors 1507 as the corrected optical axis vector 1510, and the virtual imaging information distance calculation unit 1511 outputs the subject three-dimensional coordinate value 1504 and Two corrected optical axis vectors 1510 and two lens center coordinate values 1506 from the virtual camera information input unit are input and calculated as imaging information from two virtual lenses with the two corrected optical axes as the line-of-sight directions. Composite image data 1512 and, if necessary, 2-channel pixel point distance data 1513 are calculated and output.

[Eleventh embodiment]
Next, an embodiment of the camera application device of the present invention will be described.

  In the following description of the present embodiment, the time unit of the camera applied device is indicated by a frame unit that is an image acquisition unit in order to clarify the embodiment.

FIG. 16 is a block diagram of a camera-applied device according to an eleventh embodiment of the present invention relating to claim 9. In FIG. 16, imaging information 1602 from the camera 1601 is input to the background image calculation control unit 1603 and the moving object person extraction unit 1608 in units of frames. When the background reset signal 1604 is ON, the background image calculation control unit 1603 outputs the imaging information 1602 at that time without calculating anything as the next background image 1605, and also stores it in the background image storage unit 1607. The moving object person accompaniment extraction unit 1608 inputs the imaging information 1602 and the background image storage unit 1607 as the current background image 1609, extracts and calculates only the information within the edge after the edge extraction, and calculates the moving object such as a car or a person and the vehicle. Is output as moving object person accompaniment information 1610 which is information including the accompaniment held by the person and the accompaniment held by the person.

  This calculation is performed for each frame as long as the background reset signal 1604 is turned on since the background reset signal 1604 is turned on, and the background reset signal 1604 is turned off. However, the time for performing other processing within the same frame time (hereinafter referred to as the remaining time) To achieve high-speed processing.

  When the background reset signal 1604 is off, the background image calculation control unit 1603 captures image information 1602, the current background image 1609 from the background image storage unit 1607, and the moving object person incidental information 1610 from the moving object person incidental extraction unit 1608. The following background is updated by overwriting the background image with an image of all or part of the appendage that can be determined that the appendage has surely left the moving object or person within the remaining time. The image is output as an image 1605 and stored in the background image storage unit 1607. Hereinafter, the process of overwriting the accompanying part withdrawing determination part in the background is referred to as an accompanying part leaving process.

  With the above configuration, the background image is updated as needed, and the background image storage unit 1607 leaves the background image sufficiently after the moving object such as a car or / and the accompanying object leaves the person. It can be set as the image of the state which the subsequent appendage put on the background image.

  FIG. 17 is an explanatory diagram of an attachment removal process according to an eleventh embodiment of the present invention relating to claim 9. In FIG. 17, the first line is imaging information 1602, the second line is moving person / accompanying person information 1610, the third line is current background information 1609, the fourth line is next background information 1605, and 5 frames. It shows the state of change over time, and it is determined that the accompanying object has surely left the moving object or person, for example, by performing the shape determination process of the person or car in the background image calculation control unit, It is possible to determine which of the two unconnected image area portions of frame = 4 is the accompanying object. Further, as a method of not performing the shape determination process of the person or the car, it is possible to realize a method of tracking the temporal transition of the moving object person accompaniment information and determining the area that has stopped moving as the accompaniment information.

  As described above, according to the ninth aspect of the present invention, the background image can be automatically updated from the imaging information of the fixed camera as in the eleventh embodiment.

[Twelfth embodiment]
FIG. 18 is a block diagram of a camera application apparatus according to the twelfth embodiment of the present invention relating to claim 10. In FIG. 18, the portion surrounded by a broken line has the same configuration as that of FIG. 16. Reference numerals 1801 to 1810 denote the same functions and signals as 1601 to 1610. However, when the background reset signal 1804 is turned on, the initial background image is stored in the initial background image storage unit 1820 via the background change confirmation control unit 1817, and the background is updated by the background image calculation control unit 1803. The background update signal 1811 and the accompanying information 1812 which is background image change information are output and stored in the background change storage unit 1815 simultaneously with the update time 1814 from the timer 1812. The current background information 1809 is output from the broken line.

When the background change confirmation control unit 1817 receives a background history confirmation instruction signal 1819 that is an instruction signal for confirming the background at what time in the past, the background change confirmation control unit 1817 stores the initial background image storage unit 1820. Originally, the time associated information 1816 is sequentially read from the background change storage unit 1815, and a reproduced background image 1818 up to the time designated by the background history confirmation instruction signal 1819 is generated and output.
According to the twelfth embodiment of the present invention, for example, it is possible to find a suspicious object in a security camera or store a place in a room.

[Thirteenth embodiment]
FIG. 19 is a block diagram of a camera application apparatus according to the thirteenth embodiment of the present invention relating to claim 11. 19, in-camera image storage device 1902 is provided in addition to the camera application device 1901 having the configuration shown in FIG. 16, and the next background information 1903 and the moving object human accompanying information which are the outputs of the camera application device 1901 having the configuration shown in FIG. 1904 is stored in the in-camera image storage device 1902.

  As a result, it is possible to greatly reduce and reproduce the information of the captured video in the room. Further, instead of providing the in-camera image storage device of FIG. 19, it is replaced with a host device such as a PC or a game machine, and the next background information 1903 and the moving object person accompanying information 1904 from the camera application device having the configuration of FIG. Video recording with reduced information can also be realized by a configuration in which the information is transmitted to the host and stored in a storage device in the host.

[Fourteenth embodiment]
FIG. 20 is a block diagram of a camera application apparatus according to the fourteenth embodiment of the present invention relating to claim 12. In FIG. 20, the motion information 2004 is extracted by the motion information extraction unit 2003 from the moving object person incidental information 2002, which is the output from the camera application device 2001 having the configuration shown in FIG. 16, and is transmitted to the host device 2005. With this configuration, for example, in a motion control type game using motion information, the amount of information transmitted is small, and at the same time, it is not necessary to perform motion extraction on the game machine side, so that a quick response game can be realized.

[Fifteenth embodiment]
In the fifteenth embodiment of the present invention in which the thirteenth embodiment and the fourteenth embodiment of the present invention relating to claim 13 are implemented at the same time, it is realistic in the form of reducing the burden on the game machine. It is possible to realize a game in which images are displayed on the host side and motion control is performed.

[Sixteenth embodiment]
A fifteenth embodiment of the present invention relating to claim 14 is a sixteenth embodiment of the present invention comprising a motion scoring unit and a screen display device for comparing and scoring the motion information sent to the host side and the teacher motion information. By using the embodiment, game software for a dance education game can be realized.

[Seventeenth embodiment]
FIG. 21 is a block diagram of a camera-applied device according to the seventeenth embodiment of the present invention relating to claim 15. In FIG. 21, edge information 2104 is extracted by the edge extraction unit 2103 from the moving object person incidental information 2102 that is output from the camera application device 2101 having the configuration shown in FIG. 16, and the body shape information 2106 is calculated by the body shape information calculation unit 2105. By adopting such a configuration, it is possible to measure the whole body from the omnidirectional edge information collected by rotating the person toward the camera. In addition, instead of the body shape information calculation unit in the camera, edge information may be transmitted to the host side, and body shape information may be calculated on the host side.

[Eighteenth embodiment]
FIG. 22 is a block diagram of a camera-applied device according to an eighteenth embodiment of the present invention relating to claim 16. In FIG. 22, body shape information 2202 that is output from the camera application device 2201 having the configuration shown in FIG. 21 is stored in the body shape information storage unit 2203 together with the date and time information 2207 from the timer 2206, and in response to an instruction from the body shape display control unit 2208. The body shape display instruction signal 2209 is instructed to the body shape information storage unit 2203 and the display device 2205 so that the temporal change of the body shape can be displayed, so that the child's growth record can be taken and the growth history can be confirmed. can do.

[Nineteenth embodiment]
According to a nineteenth embodiment of the present invention relating to claim 17, the camera according to claim 9 to claim 16 is the compound-eye camera of claims 1 to 8, so that a security camera or a game machine with a wide viewing angle is obtained. Camera can be realized.

  This is the end of the description of the embodiment.

  The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the invention.

a, b, c, d, e, f Camera identification symbols a1, b1, e1, f1 Lens centers a2, b2, c2, d2, e2, f2 Optical axes a3, b3, e3, f3 Image sensor α Camera horizontal Angle of view β Camera vertical angle of view Xab, Xef Average optical axis

Claims (17)

  Two cameras a and b having the same horizontal angle of view α (or vertical angle of view β) are provided, and the optical axes of the cameras a and b are set on the plane including the lens centers of the cameras a and b. A compound-eye camera characterized by being arranged so as to intersect at the same angle α (or β) as the vertical field angle β).   The same four cameras a, b, c, d having a horizontal angle of view α (or vertical angle of view β), both of the cameras a, b and cameras c, d being the compound-eye camera according to claim 1, A compound eye camera characterized in that the optical axis of camera a and the optical axis of camera c are parallel to each other on a plane including four lens centers, and the optical axis of camera b and the optical axis of camera d are parallel. .   The camera includes four identical cameras a, b, e, and f having a horizontal angle of view α and a vertical angle of view β, and both the cameras a and b and the cameras e and f are the compound-eye cameras according to claim 1. A compound eye camera, wherein Xab and Xef are arranged so as to intersect at an angle β on a plane formed by an average optical axis Xab of b and an average optical axis Xef of cameras e and f.   A compound eye camera comprising at least one compound eye camera according to any one of claims 1 to 3, and a combination of these compound eye cameras (and / or single lens cameras).   5. A compound eye camera comprising the compound eye camera according to claim 1 and an image composition unit.   6. A compound-eye camera comprising the compound-eye camera according to claim 1, further comprising a distance calculation unit that calculates a distance of an overlapping region of view angles.   A compound eye camera, a subject three-dimensional coordinate calculation unit, a virtual camera information input unit, and a virtual imaging information distance calculation unit are provided, and imaging information from the compound eye camera is input to the subject three-dimensional coordinate calculation unit, and the calculated subject three-dimensional output Coordinate information, virtual lens center coordinates and virtual optical axis vector from the virtual camera information input unit are input to the virtual imaging information distance calculation unit, and composite image data and distance data calculated by the virtual imaging information distance calculation unit are output. Thus, a compound-eye camera that simultaneously obtains a composite image and distance information corresponding to a pinhole camera at the center of the virtual lens.   The optical axis convergence point calculation unit is provided, and the two virtual optical axis vectors of the virtual 3D camera from the virtual camera information input unit and the three-dimensional coordinate value of the subject calculated by the input subject three-dimensional coordinate calculation unit are used as the optical axis convergence point. Input to the calculation unit, recalculate the two optical axis vectors by calculating the optical axis convergence point, calculate the virtual imaging information corresponding to the two lenses in the virtual imaging information distance calculation unit, and calculate the two channels 8. The compound eye camera according to claim 7, wherein composite image data and distance data are output.   A camera, a background image storage unit, a moving object person accompaniment extraction unit, and a background image calculation control unit, and includes imaging information from the camera, current moving object person accompaniment information from the moving object person accompaniment extraction unit, and the current background image. Are input to the background image calculation control unit, and when it can be determined from the moving object person accompanying information that the accompanying object has left the moving object or / and the person, the attachment determination part of the accompanying object is used as the current background image. And a background image storage unit as a next background image.   A background change storage unit, a timer, a background change confirmation control unit, and an initial background image storage unit are updated. At the same time, the background image storage unit is updated, and at the same time, the change image information different from the background at the time of update is stored together with the time series information. The background change confirmation control unit can store the background image for the specified time from the image in the initial background image storage unit and the image in the background change storage unit at any time and confirm it by regenerating the image. The camera application device according to claim 9, wherein   It has a host device or (/ and) in-camera image storage device, and transmits the initial information of the background image storage unit to the host device or (/ and) in-camera image storage device in advance. Update information is sent to the host each time it is sent, and the moving object person accompaniment information from the moving object person accompaniment extraction unit is transmitted to the host or / and the in-camera image storage device at any time to reduce the amount of data communication. 10. The camera application device according to claim 9, wherein the amount of transmission information of camera information and the capacity of the image storage device in the camera are reduced.   It is equipped with a motion extraction unit that extracts motion information from the host device and the moving object person accompaniment extraction unit, and transmits only the motion information to the host device through a communication line, thereby reducing the amount of data communication to the host device. The camera application device according to claim 9.   13. A camera application device that simultaneously realizes data transmission to the host device according to claim 11 and claim 12.   A motion scoring unit that compares and grades motion information sent to the image display device and the host side with teacher motion information on the host side, and simultaneously displays the teacher image and the camera image of the person on the display area divided by the image display device 14. The camera application device according to claim 13, wherein scoring is performed sequentially.   It has an edge extraction unit and body shape information calculation unit. Edge information is extracted by the edge extraction unit from the person image information obtained from the moving object person attachment extraction unit, and edge information in all directions obtained by rotating the person is collected. 10. The camera application device according to claim 9, wherein the feed body shape information is calculated in the camera or in the host device.   It is equipped with a display device, body information storage unit, timer, body shape display control unit, and records the time and date information at the time of body measurement at the same time as the measured body shape information, and can display temporal changes. The camera application device according to claim 15, wherein the device is a camera application device.   A camera application device comprising the compound-eye camera according to claim 1, and realizing claims 9 to 16.