JP2021144061A - Imaging device, information processing device, and imaging system - Google Patents

Abstract

To provide an imaging device capable of reducing the load of processing.SOLUTION: An imaging device includes: a body part; a detection unit including an imaging unit for imaging an object and a distance measurement unit for detecting a distance from each point on the surface of the object; an information calculation unit provided in the body part to calculate at least one of shape information and texture information of the object by using a detection result of the detection unitt; and a communication unit for transmitting a calculation result of the information calculation unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to an imaging device, an information processing device, and an imaging system.

A technique has been proposed in which an object is imaged by a plurality of imaging devices, and the obtained plurality of images are input to a computer to acquire a three-dimensional shape of the object (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2010-134546

For example, when generating a three-dimensional shape of an object from a plurality of images obtained by capturing an object using an imaging device, it is necessary to transfer all the images from the imaging device to a computer and process the images on the computer. .. In this case, since all the captured images are transferred from the imaging device to the computer, communication between the imaging device and the computer (eg, communication processing, communication data amount, etc.) may increase. The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image pickup device, an information processing device, and an image pickup system capable of reducing a communication load.

According to the first aspect of the present invention, the main body, the imaging unit for imaging the object, the detection unit including the distance measuring unit for detecting the distance from each point on the surface of the object, and the main body. Provided is an imaging device including an information calculation unit that calculates at least one of shape information and texture information of an object using the detection result of the detection unit, and a communication unit that transmits the calculation result of the information calculation unit. Will be done.

According to the second aspect of the present invention, there is provided an information processing apparatus that executes a rendering process using the calculation result transmitted from the communication unit of at least one image pickup apparatus of the first aspect.

According to the third aspect of the present invention, an image pickup system including at least one image pickup device of the first aspect and an information processing device that executes a rendering process using a calculation result transmitted from a communication unit. Provided.

According to the fourth aspect of the present invention, the plurality of image pickup devices of the first aspect and the image pickup processing device that executes the rendering process using the calculation results transmitted from the communication units of the plurality of image pickup devices. An imaging system comprising, is provided.

According to the fifth aspect of the present invention, the at least one imaging device of the first aspect and the model integration unit that generates integrated model information of the object by using the calculation result transmitted from the communication unit. An imaging system is provided.

According to a sixth aspect of the present invention, a first detection unit including a first imaging unit that images an object, a first distance measuring unit that detects a distance from each point on the surface of the object, and a first detection unit. A first imaging device including a first information calculation unit that calculates first model information including at least one of shape information and texture information of the object using the detection result of the detection unit, and a second image capturing the object. At least the shape information and texture information of the object using the detection result of the second detection unit and the second detection unit including the image pickup unit and the second distance measuring unit that detects the distance from each point on the surface of the object. A second imaging device including a second information calculation unit that calculates second model information including one, and model integration that integrates the first model information and the second model information to generate integrated model information of an object. An imaging system including an information processing apparatus having a unit is provided.

According to a seventh aspect of the present invention, a first detection unit including a first imaging unit that images an object, a first distance measuring unit that detects a distance from each point on the surface of the object, and a first detection unit. The first information calculation unit that calculates the first model information including at least one of the shape information and the texture information of the object using the detection result of the detection unit and generates the first feature point data using the first model information. A first imaging device including With the second information calculation unit that calculates the second model information including at least one of the shape information and the texture information of the object using the detection result of the unit and generates the second feature point data using the second model information. First model information and second model information using the first feature point data and the second feature point data obtained by data communication between the second image pickup device and the second image pickup device and the second image pickup device. An imaging system is provided that includes a model integration unit that integrates and generates integrated model information of an object.

According to the present invention, it is possible to provide an image pickup device, an information processing device, and an image pickup system capable of reducing a communication load.

It is a figure which shows the image pickup system which concerns on 1st Embodiment. It is a block diagram which shows the image pickup system which concerns on 1st Embodiment. It is a figure which shows the detection part which concerns on 1st Embodiment. It is a figure which shows the operation of the image pickup system which concerns on 1st Embodiment. It is a block diagram which shows the image pickup system which concerns on 2nd Embodiment. It is a figure which shows the operation of the image pickup system which concerns on 2nd Embodiment. It is a figure which shows the image pickup system which concerns on 3rd Embodiment. It is a block diagram which shows the image pickup system which concerns on 3rd Embodiment. It is a figure which shows the operation of the image pickup apparatus which concerns on 3rd Embodiment. It is a figure which shows the operation of the image pickup system which concerns on 4th Embodiment. It is a block diagram which shows the image pickup system which concerns on 5th Embodiment. It is a figure which shows the operation of the image pickup system which concerns on 5th Embodiment. It is a block diagram which shows the image pickup system which concerns on 6th Embodiment. It is a block diagram which shows the image pickup system which concerns on 7th Embodiment. It is a figure which shows the operation of the image pickup system which concerns on 7th Embodiment. It is a figure which shows the operation of the image pickup system which concerns on 8th Embodiment. It is a figure which shows the operation of the image pickup system which concerns on 9th Embodiment.

[First Embodiment]
The first embodiment will be described. FIG. 1 is a diagram showing an imaging system 1 according to the present embodiment. For example, the image pickup system 1 includes an image pickup device 2, an information processing device 3, an input device 4, and a display device 5.

The image pickup device 2 captures an object OB illuminated by, for example, light from a lighting device, light from an indoor light, natural light, or the like, and captures an image of the object OB (eg, visible light image, infrared light). Image) data is acquired. Further, the image pickup device 2 detects the distance from the image pickup device 2 to each point on the surface of the object OB, and acquires the depth information of the object OB. The depth information includes, for example, information associating the position of a point on the surface of the object OB with the distance (depth, depth) from this point to the image pickup apparatus 2. The depth information is, for example, information indicating the distribution of depth (eg, depth map) in the object OB.

The image pickup device 2 performs arithmetic processing of information on the object OB by using the data of the captured image captured by the image pickup unit 15 and the depth information detected by the distance measuring unit (distance detection unit) 16. The image pickup apparatus 2 models at least a part of the object OB by arithmetic processing and calculates model information (model data). For example, the image pickup apparatus 2 performs computer graphic processing (CG processing) on at least a part of the object OB by arithmetic processing, and calculates model information (eg, CG model data). The model information includes, for example, at least one of shape information indicating the three-dimensional shape of the object OB and texture information indicating the pattern on the surface of the object OB. Further, for example, the model information is an image space such as three-dimensional point coordinates, related information of the point coordinates, texture information of the surface defined by the point coordinates and the related information, lighting conditions of the entire image, and light source information. Includes at least one piece of information and polygon data as shape information.

For example, the object OB in FIG. 1 includes a prismatic member and a soccer ball, and the shape information includes information on the surface of the prismatic member (eg, plane) and information on the surface of the soccer ball (eg, spherical surface). include. Texture information includes, for example, characters and figures on the surface of a soccer ball (eg, black pentagon, white hexagon), patterns, information defining irregularities, specific images, and colors (eg, chromatic, achromatic). ) Includes at least one piece of information.

The image pickup apparatus 2 calculates, for example, model information of a portion (field of view region) of the object OB that enters the field of view of the image pickup apparatus 2. This model information is, for example, information of a model (hereinafter, referred to as a partial model) representing a part of a model (hereinafter, referred to as an all-around model) representing the entire circumference of the object OB. The image pickup apparatus 2 can supply at least a part of the information (eg, model information) calculated by the arithmetic processing to the external apparatus. For example, the image pickup apparatus 2 supplies at least a part of the model information to the information processing apparatus 3.

Further, for example, the image pickup apparatus 2 can output at least a part of the model information to a digital apparatus capable of inputting / outputting digital information such as a barcode or a two-dimensional code. Such a digital device can display or print digital information including at least a part of model information on a display, paper, or the like. A reader device including a reader unit (eg, an optical reader) capable of reading displayed or printed digital information can input the digital information into a storage area or the like of the own device via the reader unit. Further, the reader device may further include a rendering processing unit described later. The image pickup system 1 may be configured to include the above digital device and a reader device including a reader unit. Further, the image pickup device 2 may be configured to include the above-mentioned digital device and reader device.

The information processing device 3 includes, for example, a computer system. The input device 4 includes, for example, at least one such as a keyboard, a mouse, a touch panel, a sensor such as an acceleration sensor, a voice input device, and a touch-type pen, and is connected to the information processing device 3. The input device 4 receives input of information from a user, for example, and supplies the input information to the information processing device 3. The display device 5 includes, for example, a liquid crystal display or a touch panel type display, and is connected to the information processing device 3. The display device 5 displays an image by, for example, image data supplied from the information processing device 3.

The information processing device 3 is communicably connected to the image pickup device 2 by wire or wirelessly. For example, the information processing device 3 may be connected to the image pickup device 2 via a communication cable, or may be connected to the image pickup device 2 via an internet line. Further, the information processing device 3 may be able to communicate with the image pickup device 2 by radio waves or infrared rays.

The information processing device 3 acquires information from the image pickup device 2 by communicating with the image pickup device 2. The information processing device 3 executes a rendering process using the information (eg, model information) acquired from the image pickup device 2. For example, the information processing device 3 calculates the data of the estimated image of the object OB viewed from this viewpoint based on the setting information of the viewpoint (imaging direction) input to the input device 4 by the user. The information processing device 3 supplies, for example, the data of the estimated image to the display device 5, and causes the display device 5 to display the estimated image.

The image pickup apparatus 2 extracts, for example, information used for the rendering process of the information processing apparatus 3 from the data of the captured image and the depth information. The information extracted by the image pickup apparatus 2 is, for example, at least a part of the model information. Since the image pickup device 2 supplies the extracted information to the information processing device 3, for example, as compared with the case where the captured image data and the depth information are supplied to the information processing device 3, the image pickup device 2 and the information processing device 3 It is possible to reduce the communication load between the two and the processing load of the information processing device 3.

The image pickup device 2 may be, for example, a portable information terminal, a fixed-point camera, or a camera whose field of view can be changed manually or automatically. The image pickup system 1 may include, for example, a moving device for moving the image pickup device 2. This moving device may change the field of view of the image pickup device 2 by moving the image pickup device 2. The control unit that controls this mobile device may be provided in the image pickup system 1, for example, the image pickup device 2 or the information processing device 3. Further, the control unit that controls this mobile device may be provided in an external device of the imaging system 1.

Next, each part of the imaging system 1 will be described. FIG. 2 is a block diagram showing an imaging system 1 according to the present embodiment. The image pickup device 2 includes a detection unit 11, an information calculation unit 12, a communication unit 13, and a main body unit 14. The main body portion 14 is, for example, a camera body, a case, a housing, or the like. The detection unit 11, the information calculation unit 12, and the communication unit 13 are provided in the main body unit 14.

The detection unit 11 includes an imaging unit 15 and a distance measuring unit 16. FIG. 3A is a diagram showing an example of the detection unit 11. The imaging unit 15 includes an imaging optical system 17 and an image sensor 18. The image sensor 18 is, for example, a CMOS image sensor in which a plurality of pixels are two-dimensionally arranged, or a CCD image sensor. The image sensor 18 is housed in the main body 14. The image sensor 18 has sensitivity in, for example, the wavelength band of visible light (eg, 380 nm or more and 750 nm or less) or the wavelength band of infrared light. The imaging result of the image sensor 18 includes, for example, information on the gradation value for each color of each pixel (eg, RGB data). The image sensor 18 outputs, for example, an imaging result (detection result) in a full-color image data format. The full-color image is, for example, an image in which each of the red (R), green (G), and blue (B) of each pixel is represented by a gradation value (eg, 256 gradations).

The imaging optical system 17 includes, for example, a plurality of lenses, and forms an image of an object surface (eg, an object OB) on the image sensor 18. The imaging optical system 17 is held in the lens barrel, for example, and is attached to the main body 14 together with the lens barrel. The imaging optical system 17 and the lens barrel are, for example, interchangeable lenses, and are removable from the main body 14. The lens barrel may be a part of the main body 14, or may not be removable from the main body 14.

The distance measuring unit 16 detects the distance from each point on the surface of the object OB. The distance measuring unit 16 includes, for example, a ToF (Time of Flight) sensor, and detects the distance by the ToF method. Further, for example, the distance measuring unit 16 includes a phase difference sensor and detects the distance by the phase difference method. The ranging unit 16 includes an irradiation unit 19, an imaging optical system 20, an image sensor 21, and a controller 22.

The irradiation unit 19 can irradiate the object OB with infrared light. The irradiation unit 19 is controlled by the controller 22. The controller 22 changes the intensity of the infrared light emitted from the irradiation unit 19 over time (eg, amplitude-modulates).

The image sensor 21 is, for example, a CMOS image sensor in which a plurality of pixels are two-dimensionally arranged, or a CCD image sensor. The image sensor 21 is housed in the main body 14. The imaging optical system 20 includes, for example, a plurality of lenses, and forms an image of an object surface (eg, an object OB) on the image sensor 21. The imaging optical system 20 may be an interchangeable lens or may be built in the main body 14. The image sensor 21 has sensitivity at least in the wavelength band of the light emitted by the irradiation unit 19. For example, the image sensor 21 is controlled by the controller 22 and detects infrared light reflected and scattered by the object OB.

The controller 22 detects the distance from the surface of the object OB to the image sensor 21 by using the detection result detected by the image sensor 21 of the distance measuring unit 16. For example, when the depths of the first position and the second position on the surface of the object OB are different, the light reflected and scattered at the first position is compared with the light reflected and scattered at the second position from the object OB. The flight distance (optical path) to the image sensor 21 is different. Therefore, the phase of the light reflected and scattered at the first position of the object OB is different from the phase of the light reflected and scattered at the second position. Since the controller 22 time-modulates the intensity of the light emitted from the irradiation unit 19, the intensity of the light incident on the image sensor 21 changes depending on the phase. The controller 22 calculates the depth based on, for example, the time change of the output of the image sensor 21. The controller 22 calculates the depth from the distance between the image sensor 21 and the surface of the object OB for each partial region (eg, one pixel, a plurality of pixels) of the image captured by the image sensor 21, and calculates the depth. Associate the depth (or distance) with the position of the subregion. In this way, the controller 22 calculates, for example, the depth information including the depth of each of the plurality of subregions.

The irradiation unit 19 may irradiate light including visible light, and the image sensor 21 may detect the light reflected and scattered by the object OB. For example, the ranging unit 16 detects the object OB by the image sensor 21 in the light irradiation state and the light non-irradiation state by the irradiation unit 19, and uses the difference between the detection results in the two states to provide depth information. May be calculated.

The ranging unit 16 may include a projector camera. In this case, the ranging unit 16 uses the image sensor 21 to irradiate the object OB with light having a predetermined intensity distribution (eg, pattern light, structured light, texture) from the irradiation unit 19. Image the OB. The controller 22 detects the distance between the object OB and the distance measuring unit 16 by using the intensity distribution of the light on the object OB captured in the image captured by the image sensor 21. Further, the ranging unit 16 may include a laser scanner. In this case, the ranging unit 16 irradiates the object OB with a laser beam from the irradiation unit 19, scans the object OB with the laser beam, and detects the laser beam reflected by the object OB, thereby detecting the object OB. The distance between the distance measuring unit 16 and the distance measuring unit 16 is detected.

FIG. 3B is a diagram showing another example of the detection unit 11. The ranging unit 16 of the detection unit 11 detects the distance between the object OB and the detection unit 11 by using the detection result of detecting the object OB from a plurality of visual fields. The imaging unit 15 includes an imaging optical system 17a and an image sensor 18a, and an imaging optical system 17b and an image sensor 18b. The imaging optical system 17a and the image sensor 18a image the object OB in the first visual field. The imaging optical system 17b and the image sensor 18b image the object OB in the second visual field. The second visual field is different from the first visual field in at least one of the position of the viewpoint and the direction of the line of sight. The position of the viewpoint is a position where an image of an object is formed, such as the position of the image sensor 18a and the position of the image sensor 18b. The direction of the line of sight is, for example, parallel to the direction of the optical axis of the optical member arranged on the object side of the optical members through which the light passing toward the viewpoint passes. For example, the direction of the line of sight of the first visual field is coaxial with the optical axis on the emission side of the imaging optical system 17a. The direction of the line of sight of the second visual field is coaxial with the optical axis on the emission side of the imaging optical system 17b. The distance measuring unit 16 detects the distance between the object OB and the detection unit 11 by using the image captured by the image sensor 18a and the image captured by the image sensor 18b as a parallax image.

The image pickup unit 15 captures an image from the first visual field and an image from the second visual field at different times by one image sensor, and the distance measuring unit 16 uses these images to capture the object OB and the detection unit. The distance to 11 may be detected.

The distance measuring unit 16 may detect a distance represented by a dimensional value such as a meter, or may detect a relative distance represented by a dimensionless value standardized by a reference value. good. Further, the distance measuring unit 16 may be in a unitized form, for example, or may be externally attached to the main body unit 14. Further, at least a part of the optical system included in the distance measuring unit 16 may be shared with at least a part of the optical system included in the imaging unit 15. For example, in FIG. 3A, the distance measuring unit 16 branches infrared light among the light passing through the imaging optical system 17 with a dichroic mirror or the like, and detects the branched infrared light with an image sensor 21. It may be. Further, in the projector camera, the irradiation unit 19 may project a pattern onto the object OB via the imaging optical system 17.

Returning to the description of FIG. 2, the information calculation unit 12 includes, for example, a digital signal processor (DSP). The information calculation unit 12 calculates at least one of the shape information and the texture information of the object OB by using the detection result of the detection unit 11.

The information calculation unit 12 calculates the coordinates of a plurality of points on the surface of the object OB (hereinafter referred to as point cloud data) and the surface information including the connection information between the plurality of points as the shape information. The surface information is, for example, polygon data, vector data, draw data, or the like. The connection information includes, for example, information relating the points at both ends of the line corresponding to the ridge line (eg, edge) of the object OB to each other, and information relating a plurality of lines corresponding to the contour of the surface of the object OB to each other.

First, the information calculation unit 12 calculates the point cloud data using the detection result (eg, depth information) of the distance measuring unit 16 (point cloud data processing). For example, the information calculation unit 12 calculates the point cloud data by fluoroscopic conversion from the distance image indicated by the depth information to the plane image. When the field of view is different between the image pickup unit 15 and the distance measurement unit 16, the information calculation unit 12 changes the detection result of the distance measurement unit 16 from the field of view of the image pickup unit 15 by fluoroscopic transformation (projection transformation) or the like. The OB may be converted into the detected result. Even if the information calculation unit 12 executes fluoroscopic conversion using parameters that depend on the positional relationship between the field of view of the imaging unit 15 and the field of view of the distance measuring unit 16 (eg, the position of the viewpoint, the direction of the line of sight). good.

Then, the information calculation unit 12 estimates, for example, a surface between a point selected from a plurality of points included in the point cloud data and a point in the vicinity thereof, and uses the point cloud data as a polygon having plane information between the points. Convert to data (surface processing). The information calculation unit 12 converts the point cloud data into polygon data by, for example, an algorithm using the least squares method. This algorithm may be, for example, an application of an algorithm published in a point cloud processing library.

Next, the information calculation unit 12 calculates the texture information by, for example, an inverse rendering method. The texture information includes, for example, pattern information indicating a pattern on the surface of the object OB, light source information of light illuminating the object OB, and optical characteristics indicating the optical characteristics (eg, reflectance, scattering rate) of the surface of the object OB. Includes at least one item of information. The light source information includes, for example, information on at least one item of the position of the light source, the direction in which the light is emitted from the light source to the object, the wavelength of the light emitted from the light source, and the type of the light source.

The information calculation unit 12 calculates the light source information by using, for example, a model assuming Lambertian reflection, a model including Albedo estimation, and the like. For example, the information calculation unit 12 estimates, among the pixel values of each pixel of the image captured by the imaging unit 15, a component derived from the light diffused by the object OB and a component that is specularly reflected by the object OB. Further, the information calculation unit 12 calculates the direction in which light is incident from the light source to the object OB by using, for example, the estimation result of the component that is specularly reflected by the object OB and the shape information. The information calculation unit 12 estimates the reflection characteristic of the object OB by using, for example, the calculated light source information and the shape information, and calculates the optical characteristic information including the estimation result of the reflection characteristic. Further, the information calculation unit 12 calculates pattern information by removing the influence of illumination light from the data of the visible light image by using, for example, the calculated light source information and optical characteristic information.

Here, the image pickup apparatus 2 according to the present embodiment further includes a display unit 25, an input unit 26, a storage unit 27, and a control unit 28. The display unit 25 (see FIG. 1) is, for example, a liquid crystal display or a touch panel type display provided on the main body unit 14. The display unit 25 displays, for example, at least a part of the detection result of the detection unit 11 (eg, visible light image by the imaging unit 15), the calculation result of the information calculation unit 12 (eg, depth map), and various setting information. ..

The input unit 26 is, for example, an operation button provided on the main body unit 14, a touch panel provided on the display unit 25, a voice input device for recognizing a voice of a user or the like. The input unit 26 detects an operation by the user, for example, and accepts an input of information from the user. The input unit 26 transmits the input information to the control unit 28.

The storage unit 27 is, for example, a non-volatile memory such as a USB memory or a memory card, and stores various information. The storage unit 27 may include a storage device built in the image pickup device 2, or may include a port to which a storage device that can be released from the image pickup device 2 can be connected. Further, the information calculation unit 12 of the image pickup apparatus 2 in the present embodiment generates model information in which header information (identification information such as a number and a code) is added to information including at least one of shape information and texture information. Then, the communication unit 13 transmits the model information. The header information includes identification information, position (position information) of the imaging device 2, imaging timing by the imaging unit 15, imaging time by the imaging unit 15, optical characteristic information of the object OB, and imaging environment information (eg, light source). It includes at least one of information, lighting conditions for the object OB, etc.). In this way, the information calculation unit 12 of the image pickup apparatus 2 can generate model information having header information based on a predetermined data format, and the communication unit 13 can transmit the model information.

The control unit 28 controls each unit of the image pickup device 2 by, for example, a command (control signal) from a user or an external device (eg, information processing device 3). For example, the control unit 28 causes the detection unit 11 to execute the above detection process. This detection process includes, for example, an image pickup process by the imaging unit 15 and a distance detection process by the distance measuring unit 16. The control unit 28 stores, for example, at least a part of the detection result of the detection unit 11 in the storage unit 27. The control unit 28 causes, for example, the information calculation unit 12 to calculate the model information. The control unit 28 stores, for example, at least a part of the model information calculated by the information calculation unit 12 in the storage unit 27.

The storage unit 27 stores, for example, model information for each item. For example, the storage unit 27 stores information of each item of shape information, texture information, light source information, optical characteristic information of the object OB, and pattern information in individual data tables. The control unit 28 causes the display unit 25 to display, for example, an image showing at least a part of the information stored in the storage unit 27. The control unit 28 controls the communication unit 13 to transmit information and receive information via the communication unit 13.

The communication unit 13 includes, for example, an I / O port such as a USB port, and at least one of a communication device that performs radio wave or infrared wireless communication. The communication unit 13 is controlled by the control unit 28, reads out the information stored in the storage unit 27, and transmits the read information to the external device. For example, the communication unit 13 transmits at least a part of the calculation result (eg, model information) of the information calculation unit 12 to the information processing device 3. Further, the communication unit 13 receives, for example, information including a command from an external device. The communication unit 13 can store the received information in the storage unit 27 and supply the received information to the control unit 28. When the image pickup apparatus 2 includes the above-mentioned digital apparatus, the communication unit 13 may transmit at least a part of the model information to the digital apparatus. Then, the digital device may generate digital information based on the received model information and output the digital information to a medium such as paper.

As shown in FIG. 2, the information processing device 3 includes a communication unit 30, a storage unit 31, a rendering processing unit 32, and a control unit 33. The communication unit 30 includes, for example, at least one of a USB port, a network card, and a communication device that performs radio wave or infrared wireless communication. The communication unit 30 can communicate with the communication unit 13 of the image pickup device 2.

The storage unit 31 includes, for example, a removable storage medium such as a USB memory, and a large-capacity storage device such as an external or internal hard disk. The storage unit 31 stores, for example, at least a part of the data of the information received via the communication unit 30, an image pickup control program that controls the image pickup device 2, a processing program that executes each process of the information processing device 3, and the like. ..

The rendering processing unit 32 includes, for example, a graphics processing unit (GPU). The rendering processing unit 32 may have a mode in which the CPU and the memory execute each processing according to the image processing program. The rendering processing unit 32 performs at least one of drawing processing, texture mapping processing, and shading processing, for example.

In the drawing process, the rendering processing unit 32 can calculate, for example, an estimated image (eg, a reconstructed image) of the shape defined in the shape information of the model information viewed from an arbitrary viewpoint. In the following description, the shape indicated by the shape information is referred to as a model shape. The rendering processing unit 32 can reconstruct the model shape (example, estimated image) from the model information (example, shape information) by, for example, drawing processing. The rendering processing unit 32 stores, for example, the calculated estimated image data in the storage unit 31. Since the image pickup apparatus 2 can transmit at least a part of the model information to the information processing apparatus 3, for example, the information processing apparatus 3 can reduce the load of the rendering process. Further, for example, the imaging device 2 does not need to transmit all the captured images captured by the imaging unit 15 to the information processing device 3, and at least a part of the model information calculated by the information calculation unit 12 (eg, shape information). And texture information) is transmitted to the information processing device 3. Therefore, the imaging device of the present embodiment can reduce the communication load of information required for the drawing process of the rendering processing unit 32.

Further, in the texture mapping process, the rendering processing unit 32 can calculate, for example, an estimated image in which the image indicated by the texture information of the model information is pasted on the surface of the object on the estimated image. The rendering processing unit 32 can also calculate an estimated image in which a texture different from the object OB is attached to the surface of the object on the estimated image.

In the shading process, the rendering processing unit 32 can calculate, for example, an estimated image in which the shadow formed by the light source indicated by the light source information of the model information is added to the object on the estimated image. Further, in the shading process, the rendering processing unit 32 can calculate, for example, an estimated image in which a shadow formed by an arbitrary light source is added to an object on the estimated image.

The control unit 33 controls, for example, each unit of the information processing device 3, an image pickup device 2, an input device 4, and a display device 5. The control unit 33 controls, for example, the communication unit 30 and causes the image pickup device 2 to transmit a command (control signal) and setting information. The control unit 33 stores, for example, the information received from the image pickup device 2 by the communication unit 30 in the storage unit 31. The control unit 33 controls, for example, the rendering processing unit 32 to execute the rendering process.

The control unit 33 controls the image pickup device 2 by sending a command (signal) to the image pickup device 2 via the communication unit 30, for example. For example, the control unit 33 controls the communication unit 30 and transmits a command (request signal) requesting the transmission of predetermined information to the image pickup apparatus 2. The control unit 33 may transmit a command for causing the image pickup device 2 to execute each process to the image pickup device 2. For example, the control unit 33 may transmit a command to the detection unit 11 of the image pickup device 2 to execute the detection process to the image pickup device 2. For example, the control unit 33 may transmit a command to the information calculation unit 12 of the image pickup device 2 to execute the calculation process of the model information to the image pickup device 2. Further, the control unit 33 of the information processing device 3 transmits a command for causing the image pickup unit 15 of the image pickup device 2 to perform imaging of the object OB and a command for setting the imaging conditions for the object OB to the image pickup device 2. May be good.

For example, the communication unit 13 selectively transmits the information calculated by the information calculation unit 12 for each item. For example, the setting information stored in the storage unit 27 includes transmission item information that determines whether or not to transmit information of each item of model information, and transmission order information that defines the order in which information of each item is transmitted. include. This setting information can be updated by, for example, an operation of the input unit 26, a command from the information processing device 3, or the like. The control unit 28 controls, for example, the communication unit 13 to transmit the information of the items specified in the transmission item information in the order specified in the transmission order information. The control unit 28 may control the communication unit 13, for example, to transmit item information (eg, shape information and texture information) defined in the transmission item information at once based on a predetermined data format. good.

The transmission item information may be set, for example, depending on whether or not it is used for the rendering process of the information processing apparatus 3. For example, in the rendering process, a texture different from that of the object OB may be combined with the shape of the object OB. In this case, the information processing device 3 can execute the rendering process without using, for example, the shape information of the object OB and the texture information of the object OB. The transmission item information is set to, for example, information that defines that shape information is transmitted but texture information is not transmitted. In this case, the information calculation unit 12 does not have to calculate the texture information.

Further, for example, in the rendering process, an image in which the illumination of the object OB is changed may be calculated. The information processing device 3 can use, for example, the shape information, the pattern information, and the optical characteristic information of the object OB, and can execute the rendering process without using the light source information. In this case, the transmission item information is set to, for example, information that determines that shape information, pattern information, and optical characteristic information are transmitted and light source information is not transmitted. Further, for example, the transmission item information is set to information that defines that at least one of shape information, texture information, pattern information, light source information, and optical characteristic information is transmitted. The image pickup apparatus 2 can reduce the communication load, for example, when transmitting a part of the model information.

The transmission order information may be set, for example, according to the priority in the rendering process of the information processing apparatus 3. For example, the transmission order information may be set to send the information of the item used first in the rendering process first. For example, in the rendering process, an image of an object OB without a texture may be calculated while changing the viewpoint, and after the viewpoint is determined, an image of an object OB with a texture may be calculated from this viewpoint. .. The information processing device 3 can calculate, for example, an image of an object OB without a texture viewed from a different viewpoint without using the shape information and the texture information. The transmission item information is set to, for example, information that defines that the shape information is transmitted first and the texture information is transmitted after the shape information. For example, when the image pickup apparatus 2 transmits information of each item of model information in an order according to the priority in the rendering process of the information processing apparatus 3, the information is transmitted in parallel with a part of the rendering process of the information processing apparatus 3. Can be sent. Further, for example, the control unit 28 of the image pickup apparatus 2 can execute at least a part of the calculation of model information by the information calculation unit 12 and the information transmission process by the communication unit 13 in parallel.

The control unit 33 of the information processing device 3 stores, for example, the information input to the input device 4 in the storage unit 31. The information input to the input device 4 includes, for example, setting information of the rendering process. This setting information includes, for example, data to be drawn (eg, shape information of model information), viewpoint information in drawing process, data of an object to which a texture is pasted in texture mapping process, and texture to be pasted in texture mapping process. It includes at least one of information (eg, texture information of model information) and information of a light source in shading processing (eg, light source information of model information). The rendering processing unit 32 executes the rendering processing according to, for example, the setting information.

The control unit 33 causes the display device 5 to display, for example, images showing various information stored in the storage unit 31. For example, the control unit 33 displays the setting information of the rendering process on the display device 5, and receives the change of the setting information by the input device 4. Further, the control unit 33 causes the display device 5 to display an image indicated by the estimated image data stored in the storage unit 31.

The information processing device 3 does not have to display the estimated image obtained by the rendering process on the display device 5, and in this case, the imaging system 1 may not include the display device 5. For example, the information processing device 3 transmits at least a part of the estimated image data calculated by the rendering process to another device (reproduction device) via the communication unit 30, and the other device displays this image. May be good. For example, the information processing device 3 may transmit the data of the estimated image to the communication unit 13 of the image pickup device 2 via the communication unit 30, and the image pickup device 2 may transmit the data of the estimated image received via the communication unit 13. The estimated image may be displayed on the display unit 25 based on the data. For example, the playback device acquires the information (estimated image) calculated by the rendering process and displays the information on the display unit.

The information processing device 3 may receive various setting information from another device via the communication unit 30, and in this case, the image pickup system 1 may not include the input device 4. For example, the imaging device 2 may transmit the setting information of the rendering process (eg, the information of the viewpoint that is the source of the estimated image) to the communication unit 30 of the information processing device 3 via the communication unit 13. Further, the information processing device 3 may execute the rendering process according to the setting information of the rendering process received from the imaging device 2.

The imaging device 2 may transmit a command for requesting estimated image data calculated by the rendering process to the communication unit 30 of the information processing device 3 via the communication unit 13. The information processing device 3 may transmit estimated image data to the communication unit 13 of the image pickup device 2 via the communication unit 30 as a response to a command from the image pickup device 2. The image pickup apparatus 2 may transmit the above-mentioned request command as a part of the setting information of the rendering process, or may transmit it as a command (eg, a control signal) different from the setting information of the rendering process.

When the image pickup device 2 is controlled by the information processing device 3 to execute the above-mentioned various processes, the image pickup device 2 does not have to include at least a part of the control unit 28, the storage unit 27, the display unit 25, and the input unit 26. good. Further, the image pickup apparatus 2 does not have to be controlled by the information processing apparatus 3 by executing the above-mentioned various processes by the user's operation, for example. For example, the image pickup apparatus 2 may execute the above-mentioned various processes without receiving a command (control signal) from an external device, and executes the above-mentioned various processes according to a user operation or a predetermined processing schedule. You may. The image pickup device 2 may transmit the calculation result (eg, model information, etc.) of the information calculation unit 12 to another device of the information processing device 3 via the communication unit 13.

Next, an image processing method will be described based on the operation of the imaging system 1. FIG. 4 is a flowchart showing the operation of the imaging system 1. In step S1 of FIG. 4, the imaging device 2 images the object OB by the imaging unit 15 and acquires the data of the captured image (for example, visible light image) (imaging process). Further, in step S2, the image pickup apparatus 2 acquires point cloud data indicating the three-dimensional shape of the surface of the object OB (point cloud data processing). In step S2, the image pickup apparatus 2 detects the distance between the object OB and the image pickup apparatus 2 by the distance measuring unit 16 and acquires the depth information (step S3). Further, the information calculation unit 12 of the image pickup apparatus 2 calculates the point cloud data of the depth information by, for example, perspective transformation (step S4). Further, in step S4, for example, the information calculation unit 12 adjusts the parallax between the visible light image data and the depth information as necessary (viewpoint conversion process). The information calculation unit 12 may perform the process of step S2 in parallel with at least a part of the process of step S1, or may perform the process of step S1 before or after the process starts.

In step S5, the information calculation unit 12 calculates (generates) model information (model information generation process). The information calculation unit 12 calculates surface information as shape information, for example (step S6) (surface processing). Further, the information calculation unit 12 calculates the texture information by the inverse rendering method using, for example, the calculated surface information (step S7). In step S7, the information calculation unit 12 calculates the light source information (step S8). Further, the information calculation unit 12 calculates pattern information using, for example, the calculated light source information (step S9). In step S5, the information calculation unit 12 stores the calculated information in the storage unit 27 for each item.

In step S10, the image pickup apparatus 2 transmits at least a part of the model information stored in the storage unit 27 to the information processing apparatus 3 via the communication unit 13. The information processing device 3 executes a rendering process using, for example, the model information received from the image pickup device 2, and displays the result (eg, an estimated image) on the display device 5.

[Second Embodiment]
Next, the second embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. FIG. 5 is a diagram showing an imaging system 1 according to the present embodiment. The image pickup apparatus 2 according to the present embodiment further includes a model integration unit 35 and a position detection unit 36.

The model integration unit 35 of the image pickup apparatus 2 detects the first model information calculated based on the result of detecting the object OB from the first direction (first detection result) and the object OB from the second direction. The integrated model information is generated by integrating with the second model information calculated based on the result (second detection result). In the present embodiment, the information calculation unit 12 calculates the first model information and the second model information by using the technique as described above.

For example, the model integration unit 35 associates the first feature point of the shape indicated by the first model information with the second feature point of the shape indicated by the second model information, and associates the first model information with the first feature point. 2 Integrate with model information. The first feature point is a portion of the shape indicated by the first model information that can be distinguished from other portions. For example, a portion defined as a surface in surface information can be distinguished from other surfaces by the shape of the outer circumference thereof. The second feature point is a portion of the shape indicated by the second model information that can be distinguished from other portions.

For example, the model integration unit 35 detects the first feature point using at least one of the shape information and the texture information included in the first model information, and detects the position of the first feature point and the feature amount of the feature point ( The first feature point data indicating the first feature amount) is calculated. The model integration unit 35 stores, for example, the first feature point data in the storage unit 27 as information of one item of the first model information. Further, the model integration unit 35 detects the second feature point using at least one of the shape information and the texture information included in the second model information, and detects the position of the second feature point and the feature of the feature point. The second feature point data indicating the quantity (second feature quantity) is calculated. The model integration unit 35 stores, for example, the second feature point data in the storage unit 27 as information of one item of the second model information. The model integration unit 35 calculates feature points and feature quantities by, for example, the SIFT (Scale-Invariant Feature Transform) method or the SURF (Speeded Up Robust Features) method. For example, the above-mentioned feature point data (eg, first feature point data, second feature point data) is data including at least the feature amount of the feature point, and is a feature of the feature point and the feature amount. The data may include only the quantity.

The model integration unit 35 matches and integrates the first feature point data having the first feature amount of the first model information and the second feature point data having the second feature amount of the second model information, and integrates the third feature point. Calculate integrated model information that indicates the data. The third feature point data is, for example, data obtained by matching the first feature point data (eg, first feature amount) and the second feature point data (eg, second feature amount). The third feature point data includes, for example, a portion of the object OB where the first feature point data and the second feature point data overlap, and models a wider range than both the first partial shape and the second partial shape. Ringed integrated data.

For example, the image pickup apparatus 2 detects the object OB from one side (one side) of an arbitrary first direction, calculates the first model information, detects the object OB from the opposite side of the first direction, and detects the object OB. When the two model information is calculated, the model integration unit 35 can calculate the integrated model information covering almost the entire circumference of the object OB. The shape indicated by the integrated model information may be a partial model indicating a part of the object OB.

Here, the integrated model information includes, for example, at least one of shape information and texture information. The shape information of the integrated model information is, for example, information that integrates the shape information of the first model information and the shape information of the second model information. Further, the texture information of the integrated model information is, for example, information that integrates the texture information of the first model information and the texture information of the second model information.

Further, the position detection unit 36 detects the position information of the image pickup apparatus 2 (eg, the detection unit 11) by, for example, a command (detection request signal) from the control unit 28. The position detection unit 36 includes, for example, at least one of a direction sensor, a GPS sensor, and an acceleration sensor. The position detection unit 36 is provided inside the main body unit 14, for example, but may be externally attached to the main body unit 14. The position detection unit 36 may be a unit that can be removed from the main body unit 14.

The position information of the detection unit 11 includes, for example, at least one of the detection direction of the detection unit 11 and the global positioning information. The detection direction of the detection unit 11 is, for example, the direction of the optical axis of the imaging optical system 17 of the imaging unit 15 shown in FIG. 3A. The detection direction of the detection unit 11 is determined in advance with respect to the posture of the image pickup device 2, for example, and the position detection unit 36 detects the posture of the image pickup device 2 by the orientation sensor to detect the detection direction of the detection unit 11. Is detected. Further, the detection unit 11 detects the global positioning information by, for example, a GPS sensor. Further, the position information of the detection unit 11 includes the amount of change in the position of the detection unit 11. The position detection unit 36 can detect the speed of the detection unit 11, for example, by time-integrating the detection result of the acceleration sensor. Further, the position detection unit 36 detects, for example, the amount of change in the position of the detection unit 11 by time-integrating the speed of the detection unit 11. For example, the detection unit 11 may detect the position information of the first accuracy by the GPS sensor, and may acquire the position information of the second accuracy higher than the first accuracy by the detection result of the acceleration sensor.

For example, when the detection unit 11 detects the object OB, the control unit 28 causes the position detection unit 36 to detect the position information of the detection unit 11. For example, the control unit 28 stores the detection result of the detection unit 11 in the storage unit 27 in association with the detection result of the position detection unit 36 (position information of the detection unit 11). The information calculation unit 12 calculates model information based on the detection result of the object OB by the detection unit 11, associates the information of each item of this model information with the position information of the detection unit 11, and stores it in the storage unit 27. .. The model information may include, for example, the position information of the detection unit 11.

The position information of the detection unit 11 is used, for example, for integrating the first model information and the second model information. For example, the model integration unit 35 receives the first position information of the detection unit 11 when the detection unit 11 detects the original data of the first model information and the detection unit 11 when the detection unit 11 detects the original data of the second model information. The first model information and the second model information may be matched and integrated by using the second position information of the detection unit 11. For example, the model integration unit 35 uses the first position information and the second position information to calculate the positional relationship between the first partial shape indicated by the first model information and the second partial shape indicated by the second model information. May be good. When the above position information is used for the integration of model information, the imaging device 2 or the information processing device 3 that executes the integration can reduce the amount of calculation required for the integration and can also reduce the mismatch in the integration. Is.

The model integration unit 35 may integrate the first model information and the second model information by using the position information of the detection unit 11 and the feature point data including the feature amount. Further, the model integration unit 35 may integrate the first model information and the second model information without using the position information of the detection unit 11 and the feature point data. For example, the model integration unit 35 may integrate the first model information and the second model information without using the feature point data (eg, feature amount) and the position information of the detection unit 11.

By the way, the region adjacent to the end of the shape indicated by the shape information of the first model information (hereinafter, referred to as an out-of-field region) includes, for example, a region that cannot be detected from the first direction. The out-of-field region is, for example, an unclosed portion in the shape indicated by the first model information. The information calculation unit 12 or the model integration unit 35 may calculate the detection direction of the detection unit 11 that can acquire the shape information of the out-of-field region. The image pickup apparatus 2 may display this detection direction on the display unit 25 as a candidate for the second direction, for example, after the detection process of the object OB from the first direction is completed.

Next, an image processing method will be described based on the operation of the imaging system 1. FIG. 6 is a flowchart showing the operation of the imaging system 1. The image pickup apparatus 2 performs the process of step S11 to acquire at least the data of the captured image in the state where the detection direction of the detection unit 11 is set to the first direction. In step S11, the imaging device 2 detects the object OB from the first direction by the detection unit 11, and acquires, for example, visible light image data and point cloud data. In step S12, the information calculation unit 12 calculates the first model information using the detection result of the detection unit 11 in step S11. In step S13, the information calculation unit 12 stores the first model information in the storage unit 27.

After the detection process by the detection unit 11 is completed in step S11, the image pickup device 2 is moved by, for example, a user's manual or a moving device (not shown), and the detection direction of the detection unit 11 is set to a second direction different from the first direction. Will be done. In step S14, the imaging device 2 detects the object OB from the second direction by the detection unit 11, and acquires, for example, visible light image data and point cloud data. In step S15, the information calculation unit 12 calculates the second model information using the detection result of the detection unit 11 in step S14. In step S16, the information calculation unit 12 stores the second model information in the storage unit 27.

In step S17, the model integration unit 35 integrates the first model information and the second model information to calculate the integrated model information. In step S17, the model integration unit 35 reads at least one of the shape information and the texture information of the first model information from the storage unit 27, and detects the first feature point using at least one of the shape information and the texture information. (Step S18). Then, the model integration unit 35 calculates the first feature point data including at least the feature amount of the first feature point (step S19), and stores the first feature point data in the storage unit 27 (step S20). The model integration unit 35 reads at least one of the shape information and the texture information of the second model information from the storage unit 27, and detects the second feature point using at least one of the shape information and the texture information (step S21). .. Then, the model integration unit 35 calculates the second feature point data including at least the feature amount of the second feature point (step S22), and stores the second feature point data in the storage unit 27 (step S23). The model integration unit 35 reads the first feature point data and the second feature point data from the storage unit 27, and matches these feature point data (step S24). The model integration unit 35 calculates the integration model information by the matching process in step S24, and stores, for example, the integration model information in the storage unit 27 for each item (step S25). Then, in step S26, the imaging device 2 transmits at least a part of the integrated model information stored in the storage unit 27 to the information processing device 3 by the communication unit 13. The information processing device 3 executes a rendering process using, for example, the integrated model information received from the image pickup device 2, and displays the result (eg, estimated image, etc.) on the display device 5.

[Third Embodiment]
Next, the third embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. FIG. 7 is a diagram showing an imaging system 1 according to the present embodiment. The imaging system 1 includes a plurality of imaging devices 2. In the following description, of the plurality of image pickup devices 2, one image pickup device 2 is referred to as a first image pickup device 2a, and the other one image pickup device 2 is referred to as a second image pickup device 2b. Therefore, the imaging system 1 is configured to include at least two imaging devices or a plurality of imaging devices.

The first imaging device 2a detects the object OB from the first direction, and calculates the first model information based on the detection result. The first imaging device 2a supplies the first model information to the information processing device 3. The second imaging device 2b detects an object from the second direction and calculates the second model information based on the detection result. The second imaging device 2b supplies the second model information to the information processing device 3. The information processing device 3 integrates the first model information and the second model information, and is an integrated model based on the detection results detected from a plurality of directions (eg, the first direction and the second direction different from the first direction). Calculate the information.

Next, each part of the imaging system 1 will be described. FIG. 8 is a block diagram showing the imaging system 1. The first imaging device 2a is referred to as a time measuring unit 41 (hereinafter referred to as a time measuring unit 41a) for detecting timing information (hereinafter referred to as timing information) when the detection unit 11 (hereinafter referred to as a detecting unit 11a) detects an object OB. ) Is provided. The timing information includes, for example, information indicating a period (eg, date and time) during which the detection unit 11a has performed the detection process. The timing information may include at least one of the start time and the end time of the period in which the detection unit 11a has performed the detection process.

The timing information is used, for example, when performing model integration using a plurality of detection results having different timings when the object OB is detected by the imaging device. For example, a plurality of model information may be calculated based on at least two detection results in which the timing of detecting the object OB is relatively close among the plurality of detection results, and the calculated plurality of model information may be integrated.

For example, the detection unit 11a stores the detection result in the storage unit 27 (hereinafter, referred to as the storage unit 27a) in association with the timing information. The information calculation unit 12 (hereinafter referred to as an information calculation unit 12a) calculates the first model information based on the detection result of the detection unit 11a, and stores the calculation result in the storage unit 27a in association with the timing information. ..

Further, the second imaging device 2b has, for example, the same configuration as the first imaging device 2a. The detection unit 11 (hereinafter referred to as the detection unit 11b) of the second imaging device 2b detects the object OB from the second direction, and the time measuring unit 41 (hereinafter referred to as the time measuring unit 41b) reports the detection result. It is stored in the storage unit 27 (hereinafter referred to as the storage unit 27b) in association with the detected timing information. The information calculation unit 12 (hereinafter referred to as the information calculation unit 12b) of the second imaging device 2b calculates the second model information based on the detection result of the detection unit 11b, and associates the calculation result with the timing information. It is stored in the storage unit 27b.

The timing information may include information on a period during which the detection unit 11 is scheduled to detect the object OB. The first imaging device 2a and the second imaging device 2b may transmit and receive timing information, and may detect the object OB in synchronization with each other by using the control unit 28 and the communication unit 13.

In the present embodiment, the information calculation unit 12a of the first image pickup apparatus 2a calculates the first additional information used for integrating the first model information and the second model information. The first additional information includes, for example, information obtained by matching feature point data (eg, feature amount), position information of the detection unit 11a, timing information of the image pickup device 2, position information of the detection unit 11b, and a second image pickup device. It includes at least one of the timing information of 2b. For example, the first imaging device 2a receives the second model information from the second imaging device 2b, and the information calculation unit 12a of the first imaging device 2a receives the calculated first feature point data and the second imaging device 2b. The first additional information is calculated using the received second feature point data.

In the present embodiment, the information processing device 3 further includes a model integration unit 45. The model integration unit 45 integrates the first model information from the first image pickup apparatus 2a and the second model information from the second imaging apparatus 2b to generate the above-mentioned integrated model information. The algorithm used by the model integration unit 45 for integrating the model information may be, for example, the same as the model integration unit 35 of the image pickup apparatus 2 described in the second embodiment. The communication unit 30 of the information processing device 3 receives the first additional information from the communication unit 13a of the first imaging device 2a, for example, before the model integration is performed. The model integration unit 45 of the information processing device 2 performs model integration using, for example, the first model information, the second model information, and the first additional information.

FIG. 9 is a diagram showing the operation of the imaging system 1 according to the present embodiment. The information calculation unit 12a of the first imaging device 2a calculates the first model information in step S31, and calculates the first feature point data including at least the first feature amount in step S32. The information calculation unit 12b of the second imaging device 2b calculates the second model information in step S33, and calculates the second feature point data including at least the second feature amount in step S34. In step S35, the communication unit 13a of the first imaging device 2a transmits a command (request command) requesting transmission of the second feature point data to the communication unit 13b of the second imaging device 2b. In step S36, the communication unit 13b of the second imaging device 2b receives the request command from the communication unit 13a of the first imaging device 2a. In step S37, the communication unit 13b transmits the second feature point data to the communication unit 13a. In step S38, the communication unit 13a receives the second feature point data from the communication unit 13b. In this way, communication (data communication) of at least feature point data (eg, first feature point data and / or second feature point data) between image pickup devices (eg, between the first image pickup device and the second image pickup device). ), The information processing device 3 can reduce the load of integration by calculating the information required for the integration of the model information by each imaging device. The information processing apparatus 3 can also create the above-mentioned feature point data (first feature point data, second feature point data, etc.). In this case, for example, the information processing device 3 generates the first feature point data and the second feature point data using each model information received from each image pickup device or one image pickup device (eg, the first image pickup device 2a). do. Further, the generation of the above-mentioned feature point data (eg, first feature point data, second feature point data) can also be performed by the first imaging device 2a (or the second imaging device 2b). In this case, for example, the first imaging device 2a generates the second feature point data based on the second model information received from the second imaging device 2b.

The information calculation unit 12a of the first imaging device 2a matches the first feature point data and the second feature point data in step S39 (matching process of the feature portion), and shows the matching result in step S40. Calculate additional information. The first additional information includes, for example, information indicating the positional relationship between the shape indicated by the first model information and the shape indicated by the second model information. The first additional information includes, for example, the coordinates of the feature points common to the first feature data and the second feature data, and / or the feature amount common to the first feature data and the second feature data. The information calculation unit 12a stores the calculated first additional information in, for example, a storage unit 27a. The first additional information may be, for example, one item of information of the first model information.

In step S41, the communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 30 of the information processing device 3. In step S42, the communication unit 30 of the information processing device 3 receives the first additional information from the communication unit 13a.
In step S43, the information processing device 3 is assigned to the first responsible area to be in charge of the first model information from the first imaging device 2a and the second model information from the second imaging device 2b in the integrated model information. Determine and set the second area in charge. For example, the model integration unit 45 selects each responsible area so that the first responsible area and the second responsible area do not overlap.

In step S44, the control unit 33 of the information processing device 3 controls the communication unit 30 and issues a command (first request signal) for requesting transmission of the first model information in the first area in charge of the first imaging device 2a. It is transmitted to the communication unit 13a. Hereinafter, the first model information in the first area in charge is referred to as the first designated information. The first designated information is, for example, information regarding a part of the shape indicated by the first model information. The communication unit 13a of the first imaging device 2a receives the command of step S44 in step S45. In step S46, the information calculation unit 12a of the first image pickup apparatus 2a extracts the first designated information from the first model information. In step S47, the communication unit 13a of the first imaging device 2a transmits the first designated information to the communication unit 30 of the information processing device 3. In step S48, the communication unit 30 receives the first designated information from the communication unit 13a.

In step S49 after the processing of step S44, the control unit 33 of the information processing device 3 controls the communication unit 30 and issues a command (second request signal) requesting transmission of the second model information in the second area in charge. , Is transmitted to the communication unit 13b of the second imaging device 2b. Hereinafter, the second model information in the second area in charge is referred to as the second designated information. The second designated information is, for example, information regarding a part of the shape indicated by the second model information. The communication unit 13b of the second image pickup apparatus 2b receives the command of step S49 in step S50. In step S51, the information calculation unit 12b of the second image pickup apparatus 2b extracts the second designated information from the second model information. In step S52, the communication unit 13b of the second imaging device 2b transmits the second designated information to the communication unit 30 of the information processing device 3. In step S53, the communication unit 30 of the information processing device 3 receives the second designated information from the communication unit 13b. In step S54, the model integration unit 45 of the information processing device 3 integrates the first designated information that is a part of the first model information and the second designated information that is a part of the second model information, and integrates the model. Calculate the information. For example, the model integration unit 45 shares model information (eg, first model information, second model information) or feature point data (eg, first feature point data, second feature point data) among a plurality of imaging devices. Integrated model information is generated using the information (eg, additional information, designated information) obtained by (eg, information sharing by two-way communication, one-way communication). Further, for example, in order to create the above-mentioned additional information, the image pickup device 2a and the image pickup device 2b have information (eg, first model information, second model information, first feature point data, etc.) among a plurality of image pickup devices. Data communication of the second feature point data) is performed.

[Fourth Embodiment]
Next, the fourth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. The imaging system 1 according to the present embodiment has the same configuration as that of FIGS. 7 and 8. In the present embodiment, the first imaging device 2a and the second imaging device 2b each calculate additional information used for model integration. In the present embodiment, the information processing apparatus 3 checks the integrated model information and performs interpolation processing as appropriate.

FIG. 10 is a diagram showing the operation of the imaging system 1 according to the present embodiment. In FIG. 10, the same processing as in FIG. 9 is designated by the same reference numerals to simplify or omit the description thereof. The information calculation unit 12a of the first imaging device 2a calculates the first model information in step S31, and calculates the first feature point data including at least the first feature amount in step S32. The information calculation unit 12b of the second imaging device 2b calculates the second model information in step S33, and calculates the second feature point data including at least the second feature amount in step S34. In step S60, the communication unit 13a of the image pickup device 2 transmits the first feature point data to the communication unit 13b of the second image pickup device 2b. In step S61, the communication unit 13b receives the first feature point data from the communication unit 13a. In step S62, the communication unit 13b transmits the second feature point data to the communication unit 13a. In step S63, the communication unit 13a receives the second feature point data from the communication unit 13b.

In step S64, the information calculation unit 12a of the first imaging device 2a matches the first feature point data with the second feature point data, and calculates the first additional information indicating the matching result. In step S65, the communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 30 of the information processing device 3. In step S66, the communication unit 30 receives the first additional information from the communication unit 13a.

The information calculation unit 12b of the second imaging device 2b matches the first feature point data and the second feature point data in the same manner as the first imaging device 2a, and in step S67, the second additional information indicating the matching result. Is calculated. The information calculation unit 12b stores the calculated second additional information in the storage unit 27b. The second additional information is the same as the first additional information, and may be, for example, one item of information of the second model information. In step S68, the communication unit 13b of the second imaging device 2b transmits the second additional information to the communication unit 30 of the information processing device 3. In step S69, the communication unit 30 receives the second additional information from the communication unit 13b.

The imaging system 1 integrates the first model information and the second model information by, for example, the processes of steps S43 to S54 described with reference to FIG. 9. The model integration unit 45 of the information processing device 3 stores, for example, the calculated integrated model information in the storage unit 31. In step S70, the model integration unit 45 checks (determines) the consistency of the calculated integrated model information. When it is determined that the integrated model information is not consistent, the model integration unit 45 performs the following processing, for example.

In step S71, the control unit 33 of the information processing device 3 controls the communication unit 30 and transmits a command (request signal) requesting transmission of the first interpolation information to the communication unit 13a of the first imaging device 2a. The first interpolation information is, for example, information used for interpolation of integrated model information, and includes information on a region where the shape indicated by the first model information and the shape indicated by the second model information overlap. In step S72, the communication unit 13a of the first imaging device 2a receives the command of step S68. In step S73, the information calculation unit 12a of the first imaging device 2a extracts the first interpolation information from the first model information. In step S74, the communication unit 13a of the first imaging device 2a transmits the first interpolation information to the communication unit 30 of the information processing device 3. In step S75, the communication unit 30 receives the first interpolation information from the communication unit 13a.

In step S76 after step S71, the control unit 33 of the information processing device 3 controls the communication unit 30 and issues a command (request signal) requesting transmission of the second interpolation information to the communication unit of the second imaging device 2b. It is transmitted to 13b. The second interpolation information is, for example, information used for interpolation of integrated model information, and includes information on a region where the shape indicated by the first model information and the shape indicated by the second model information overlap. In step S77, the communication unit 13b of the second imaging device 2b receives the command of step S76. In step S78, the information calculation unit 12b of the second imaging device 2b extracts the second interpolation information from the second model information. In step S79, the communication unit 13b of the second imaging device 2b transmits the second interpolation information to the communication unit 30 of the information processing device 3. In step S80, the communication unit 30 receives the second interpolation information from the communication unit 13b.

In step S81, the model integration unit 45 of the information processing device 3 performs interpolation processing of the integrated model information using the first interpolation information and the second interpolation information. In step S82, the model integration unit 45 updates the integrated model information stored in the storage unit 31 with the integrated model information after the interpolation processing.

Further, for example, the model integration unit 45 may use model information (eg, first model information, second model information) or feature point data (eg, first feature point data, second feature point data) between a plurality of imaging devices. Integrated model information is generated using the information (eg, additional information, designated information) obtained by sharing the information (eg, sharing information by two-way communication, one-way communication).

[Fifth Embodiment]
Next, the fifth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. FIG. 11 is a block diagram showing an imaging system 1 according to the present embodiment.

The imaging system 1 according to the present embodiment includes a first imaging device 2a, a second imaging device 2b, and an information processing device 3. The first imaging device 2a includes, for example, a model integration unit 35 (hereinafter, referred to as a model integration unit 35a) similar to that of the second embodiment. The model integration unit 35a integrates the first model information calculated by the information calculation unit 12a and the second model information received by the communication unit 13a from the second imaging device 2b. The communication unit 13a transmits the integrated model information calculated by the model integration unit 35a to the communication unit 30 of the information processing device 3.

FIG. 12 is a diagram showing the operation of the imaging system 1 according to the present embodiment. In FIG. 12, the same processing as in FIG. 9 and the like is designated by the same reference numerals to simplify or omit the description thereof. The imaging system 1 calculates the first additional information used for model integration by the processing of steps S31 to S40.

In step S85, the first image pickup apparatus 2a has a first responsible area in charge of the first model information and a second responsible area in charge of the second model information from the second image pickup apparatus 2b in the integrated model information. Decide and set. For example, the model integration unit 35a selects each responsible area so that the first responsible area and the second responsible area do not overlap.

In step S86, the control unit 28 of the first imaging device 2a controls the communication unit 13a and transmits a command requesting transmission of the second model information in the second area in charge to the communication unit 13b of the second imaging device 2b. do. The communication unit 13b of the second image pickup apparatus 2b receives the command of step S86 in step S87. In step S88, the information calculation unit 12b of the second imaging device 2b extracts the information of the second responsible area designated from the first imaging device 2a from the second model information. In step S89, the communication unit 13b of the second imaging device 2b transmits the information (extracted information) extracted by the information calculation unit 12b in step S86 to the communication unit 13a of the first imaging device 2a.

In step S90, the information calculation unit 12a of the first imaging device 2a extracts the first model information of the first responsible area. In addition, the communication unit 13a of the first imaging device 2a receives the extraction information (eg, the second model information of the second area in charge) from the communication unit 13b in step S91. In step S92, the model integration unit 35a of the first imaging device 2a integrates the first model information of the first responsible area and the second model information of the second responsible area to calculate the integrated model information. In step S93, the communication unit 13a of the first imaging device 2a transmits the integrated model information to the communication unit 30 of the information processing device 3. In step S94, the communication unit 30 receives the integrated model information from the communication unit 13a. In step S95, the rendering processing unit 32 of the information processing device 3 performs rendering processing using the integrated model information.

Further, for example, the model integration unit 35a of the first imaging device 2a may use model information (eg, first model information, second model information) or feature point data (eg, first feature point data,) between a plurality of imaging devices. Integrated model information is generated using the information (eg, additional information, designated information) obtained by sharing the second feature point data) (eg, sharing information by two-way communication, one-way communication).

[Sixth Embodiment]
Next, the sixth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. FIG. 13 is a block diagram showing an imaging system 1 according to the present embodiment.

In the present embodiment, the first imaging device 2a and the second imaging device 2b communicate with each other the information used for model integration, and calculate the integrated model information, respectively. The first image pickup apparatus 2a calculates the integrated model information by, for example, the process described with reference to FIG. Further, the second imaging device 2b calculates the integrated model information by performing the same processing as, for example, the first imaging device 2a.

The information processing device 3 receives integrated model information from at least one of the first imaging device 2a and the second imaging device 2b, and executes a rendering process using the received integrated model information. For example, the communication unit 30 of the information processing device 3 transmits a command requesting transmission of integrated model information to the communication unit 13a of the first imaging device 2a. The control unit 28 of the first imaging device 2a controls the communication unit 13a to transmit the information of the item specified in the command among the integrated model information to the communication unit 30 of the information processing device 3. Similarly, the information processing device 3 can cause the second imaging device 2b to transmit the integrated model information.

The information processing device 3 monitors, for example, load information (eg, the ratio of the load to the processing power) of each part of the imaging system 1, and processes related to model integration in the processing unit having a relatively small load in the imaging system 1. May be assigned. For example, the control unit 28 of the first imaging device 2a generates load information of the first imaging device 2a. The load information of the first imaging device 2a includes, for example, the load information of the information calculation unit 12a and the load information of the model integration unit 35a. The communication unit 13a of the first imaging device 2a transmits the load information of the first imaging device 2a to the communication unit 30 of the information processing device 3. Similarly, the second imaging device 2b transmits the load information of the second imaging device 2b to the communication unit 30 of the information processing device 3. The control unit 33 of the information processing device 3 compares the load of each unit based on the received load information, and causes the processing unit having a relatively light load to execute processing (processing related to model integration) only on one imaging device. Alternatively, a plurality of imaging devices are made to partially share the processing (processing related to model integration) and execute the processing.

For example, when the control unit 33 determines that the load of the model integration unit 35a of the first imaging device 2a is larger than the load of the model integration unit 35b of the second imaging device 2b, the control unit 33 integrates the model by the first imaging device 2a. At least a part thereof may be restricted, and the limited processing may be executed by the second imaging device 2b. For example, the control unit 33 may prohibit the calculation of the first additional information by the first imaging device 2a and cause the second imaging device 2b to transmit the second additional information to the first imaging device 2a. 2a may perform model integration using the second additional information. Further, for example, the control unit 33 may prohibit the process of determining the area in charge by the first imaging device 2a, and cause the second imaging device 2b to transmit the area information in charge defining the area in charge from the second imaging device 2b to the first imaging device 2a. , The first image pickup apparatus 2a may perform model integration using the area information in charge. Further, for example, the control unit 33 may prohibit model integration by the first imaging device 2a and receive integrated model information from the second imaging device 2b.

When the control unit 33 determines that the load of the information processing device 3 is smaller than that of either the first image pickup device 2a or the second image pickup device 2b, at least one of the first image pickup device 2a and the second image pickup device 2b. At least a part of the model integration process of the above may be restricted, and the limited process may be executed by the model integration unit 45. For example, the control unit 33 prohibits the determination process of the area in charge by each of the first imaging device 2a and the second imaging device 2b, causes the model integration unit 45 to execute the determination process of the area in charge, and transmits the area information in charge. It may be transmitted to each of the first image pickup apparatus 2a and the second imaging apparatus 2b. Further, the control unit 33 may prohibit the model integration process by each of the first image pickup device 2a and the second image pickup device 2b, and cause the model integration unit 45 of the information processing device 3 to execute the model integration process.

A load control unit (load monitoring unit) that executes at least one of load control and monitoring of the imaging system 1 like the control unit 33 is provided in the first imaging device 2a (eg, control unit 28). It may be provided in the second imaging device 2b (eg, the control unit 28). Further, the load control unit (load monitoring unit) may or may not be provided outside the image pickup system 1.

Further, for example, the model integration unit 35a (or / and the model integration unit 35b of the second imaging device 2b) of the first imaging device 2a provides model information (eg, first model information, second model) between a plurality of imaging devices. Information (eg, additional information, designation) obtained by sharing (eg, bidirectional communication, information sharing by one-way communication) of feature point data (eg, first feature point data, second feature point data) Information) is used to generate integrated model information.

[7th Embodiment]
Next, the seventh embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. FIG. 14 is a block diagram showing an imaging system 1 according to the present embodiment.

The imaging system 1 according to the present embodiment includes a first imaging device 2a, a second imaging device 2b, and an information processing device 3. The first imaging device 2a and the second imaging device 2b can calculate integrated model information, respectively. In the present embodiment, the information processing device 3 does not include a model integration unit. The information processing device 3 receives integrated model information from at least one of the first imaging device 2a and the second imaging device 2b, and executes a rendering process using the received integrated model information.

FIG. 15 is a diagram showing the operation of the imaging system 1 according to the present embodiment. In the following description, the same processing as in FIG. 10 and the like will be designated by the same reference numerals and the description thereof will be omitted or simplified. The first imaging device 2a calculates the first model information (step S31) and calculates the first feature point data (step S32). The second imaging device 2b calculates the second model information (step S33) and calculates the first feature point data (step S34). The communication unit 13a of the first imaging device 2a transmits the first feature point data to the communication unit 13b of the second imaging device 2b (step S60), and the communication unit 13b receives the first feature point data (step S61). ). The communication unit 13b transmits the second feature point data to the communication unit 13a of the first imaging device 2a (step S62), and the communication unit 13a receives the second feature point data (step S63).

The information calculation unit 12a of the first imaging device 2a calculates the first additional information using the first feature point data and the second feature point data (step S64). The information calculation unit 12b of the second imaging device 2b calculates the second additional information using the first feature point data and the second feature point data (step S67). The communication unit 13a of the first imaging device 2a transmits the first additional information to the communication unit 13b of the second imaging device 2b (step S101), and the communication unit 13b receives the first additional information (step S102). The communication unit 13b transmits the second additional information to the communication unit 13a of the first imaging device 2a (step S103), and the communication unit 13a receives the second additional information (step S104).

The model integration unit 35a of the first imaging device 2a determines the area in charge using the first additional information and the second additional information (step S105). The information calculation unit 12a of the first imaging device 2a extracts the first model information of the first area in charge (step S106). The model integration unit 35b of the second imaging device 2b determines the area in charge using the first additional information and the second additional information (step S107). The information calculation unit 12b of the second imaging device 2b extracts the second model information of the second responsible area (step S108).

The communication unit 13a of the first imaging device 2a transmits the first model information of the first responsible area to the communication unit 13b of the second imaging device 2b (step S109), and the communication unit 13b is the first in the first responsible area. Receive model information (step S110). The communication unit 13b transmits the second model information of the second responsible area to the communication unit 13a of the first imaging device 2a (step S111), and the communication unit 13a receives the second model information of the second responsible area (step S111). Step S112).

The model integration unit 35a of the first imaging device 2a integrates the model using the first model information of the first responsible area and the second model information of the second responsible area (step S113). Then, the model integration unit 35a checks the integrated integrated model information (first integrated model information) (step S114). The model integration unit 35b of the second imaging device 2b performs model integration using the first model information of the first responsible area and the second model information of the second responsible area (step S115). Then, the model integration unit 3ba checks the integrated integrated model information (second integrated model information) (step S116).

When it is determined that the model integration unit 35a of the first imaging device 2a and the model integration unit 35b of the second imaging device 2b are not consistent with each other, for example, the following processing is performed. The information calculation unit 12a of the first imaging device 2a extracts the first interpolation information (step S117). The information calculation unit 12b of the second imaging device 2b extracts the second interpolation information (step S118).

The communication unit 13a of the first imaging device 2a transmits the first interpolation information to the communication unit 13b of the second imaging device 2b (step S119), and the communication unit 13b receives the first interpolation information (step S120). The communication unit 13b transmits the second interpolation information to the communication unit 13a of the first imaging device 2a (step S121), and the communication unit 13a receives the second interpolation information (step S122).

The model integration unit 35a of the first image pickup apparatus 2a performs interpolation processing using the first interpolation information and the second interpolation information (step S123). The model integration unit 35a updates the integrated model information stored in the storage unit 27a to the integrated model information after the interpolation processing (step S124). The model integration unit 35b of the second image pickup apparatus 2b performs interpolation processing using the first interpolation information and the second interpolation information (step S125). The model integration unit 35b updates the integrated model information stored in the storage unit 27b to the integrated model information after the interpolation processing (step S126).

The information processing device 3 receives information from the first imaging device 2a. For example, the communication unit 30 of the information processing device 3 transmits a command requesting the transmission of information used for the rendering process to the communication unit 13a of the first imaging device 2a (step S130), and the communication unit 13a is in step S130. Receive the command (step S131). The information calculation unit 12a of the first imaging device 2a extracts, for example, the information specified in the command of step S130 from the model information (first model information or integrated model information) (step S134). The communication unit 13a of the first imaging device 2a transmits the information extracted in step S134 to the communication unit 30 of the information processing device 3 (step S135), and the communication unit 30 receives the information transmitted in step S134. (Step S136).

The information processing device 3 receives information from the second imaging device 2b as needed. For example, when the information held by the second imaging device 2b is used for the rendering process, the communication unit 30 of the information processing device 3 issues a command requesting the transmission of the information used for the rendering process to the communication unit of the second imaging device 2b. It transmits to 13b (step S132), and the communication unit 13b receives the command of step S132 (step S133). The information calculation unit 12b of the second imaging device 2b extracts, for example, the information specified in the command of step S132 from the model information (second model information or integrated model information) (step S137). The communication unit 13b of the second imaging device 2b transmits the information extracted in step S137 to the communication unit 30 of the information processing device 3 (step S138), and the communication unit 30 receives the information transmitted in step S138. (Step S139).

The rendering processing unit 32 of the information processing device 3 performs rendering processing using, for example, the information received from the first imaging device 2a and the information received from the second imaging device 2b. When the information processing device 3 does not use the information held by the second imaging device 2b for the rendering process, the information processing device 3 does not have to receive the information from the second imaging device 2b. For example, the communication unit 30 of the information processing device 3 does not have to transmit a command requesting transmission of information to the communication unit 13b of the second imaging device 2b.

Further, for example, the model integration unit 35a of the first imaging device 2a may use model information (eg, first model information, second model information) or feature point data (eg, first feature point data,) between a plurality of imaging devices. Integrated model information (first integrated model information) using information (eg, additional information, designated information) obtained by sharing (eg, information sharing by two-way communication, one-way communication) of second feature point data). To generate. For example, the model integration unit 35b of the second imaging device 2b may include model information (eg, first model information, second model information) or feature point data (eg, first feature point data, second) between a plurality of imaging devices. Integrated model information (second integrated model information) is generated using the information (example, additional information, designated information) obtained by sharing (eg, bidirectional communication, information sharing by one-way communication) of feature point data). do. In this way, for example, when each imaging device images the same object OB and generates integrated model information in each imaging device, a plurality of imaging devices (eg, first imaging device 2a, second imaging device) 2b) may be configured so that the model integration of the entire object OB is distributedly processed in each imaging device to generate integrated model information (eg, first integrated model information, second integrated model information), respectively. .. In this case, for example, the integrated model information generated by distributed processing in the imaging device is the partial integrated model information of the object OB. In such a case, for example, the first integrated model information generated by the distributed processing may be a partial integrated model corresponding to 50% of the entire area of the object OB, and the first integrated model information and the second integrated model information may be used. The integrated model information may be configured to share the integrated model information of the entire object OB with each other. Then, in response to a request from a playback device such as the information processing device 3 or a device capable of rendering, the imaging device processes information necessary for rendering (eg, first integrated model information, second integrated model information). Send to 3 or a playback device.

[8th Embodiment]
Next, the eighth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. FIG. 16 is a block diagram showing an imaging system 1 according to the present embodiment.

In the present embodiment, the first image pickup device 2a, the second image pickup device 2b, and the information processing device 3 do not include the model integration unit described above. The information processing device 3 executes the first rendering process using, for example, the information received from the first imaging device 2a (eg, integrated model information), and then receives the information received from the second imaging device 2b (eg, integrated model information). The second rendering process is executed using the model information). For example, in the first rendering process, the rendering processing unit 32 generates a first estimated image in which the object OB is viewed from the first viewpoint. For example, in the second rendering process, the rendering processing unit 32 generates a second estimated image in which the object OB is viewed from the second viewpoint. The information processing device 3 requests the second imaging device 2b to transmit information when, for example, a model is missing only from the information held by the first imaging device 2a in generating the second estimated image. .. For example, the communication unit 30 of the information processing apparatus 3 may sequentially receive the information used in the second rendering process during the period in which the first rendering process is performed.

In the case of such a configuration, for example, in order for the information processing device 3 to obtain information necessary for the rendering process of the information processing device 3, the imaging system 1 in the present embodiment can receive information such as model information or additional information. Further, it is further provided with a communication unit capable of transmitting information such as the calculated integrated model information to an external device, and an information processing device (model integration processing device) including a model integration unit that performs model integration processing.

At least one of the first image pickup device 2a, the second image pickup device 2b, and the information processing device 3 may include a model integration unit. For example, the second imaging device 2b may include a model integration unit, and the first imaging device 2a and the information processing device 3 may not include the model integration unit. In this case, for example, the information processing device 3 may execute the above-mentioned first rendering process using the information received from the first imaging device 2a. Further, the second imaging device 2b may perform at least a part of the model integration process during the period when the first rendering process is performed. Further, the information processing device 3 may perform a second rendering process using the integrated model information received from the second imaging device 2b after the first rendering process.

[9th Embodiment]
Next, the ninth embodiment will be described. In the present embodiment, the same components as those in the above-described embodiment are designated by the same reference numerals to simplify or omit the description thereof. FIG. 17 is a block diagram showing an imaging system 1 according to the present embodiment.

In the present embodiment, the communication unit 13 of the image pickup device 2 can receive image data of the object OB from an external device (eg, the image pickup device 50 as the second image pickup device). The image pickup device 50 may be a part of the image pickup system 1 or an external device of the image pickup system 1. The image pickup device 50 does not include, for example, a distance measuring unit like the image pickup device 2. In this case, the image pickup device 50 transmits an image having a parallax (parallax image) among the images captured by the image pickup unit 51 to the image pickup device 2. Then, the image pickup apparatus 2 calculates the depth information of the image pickup apparatus 50 using the received parallax image. The image pickup apparatus 50 does not calculate the model information described above, for example. The image pickup device 50 is, for example, a general-purpose camera, a high-performance mobile phone such as a smartphone, a tablet, or the like.

The image pickup device 50 includes, for example, an image pickup unit 51, an input unit 52, a storage unit 53, a display unit 54, a communication unit 55, and a control unit 56. The imaging unit 51 includes, for example, a CCD image sensor and a CMOS image sensor, and can image an object OB. The storage unit 53 includes, for example, a non-volatile memory, and stores image data and various setting information captured by the image pickup unit 51. The input unit 52 receives, for example, a command from the user. The display unit 54 displays, for example, image data stored in the storage unit 53 or an image indicated by various setting information. The communication unit 55 can communicate with the communication unit 13 of the image pickup apparatus 2 by wire or wirelessly. For example, the communication unit 55 transmits the image data of the object OB imaged by the image pickup unit 51 to the communication unit 13 of the image pickup device 2. The control unit 56 controls each unit of the image pickup apparatus 50.

The information calculation unit 12 of the image pickup apparatus 2 calculates the first model information using the detection result of the detection unit 11 as described in the above-described embodiment. Further, the information calculation unit 12 of the image pickup device 2 calculates the second model information based on, for example, the image captured by the image pickup device 15 and / or the image (image data of the object OB) received from the image pickup device 50. do. The information calculation unit 12 does not have to calculate the second model information, and for example, the image received from the image pickup apparatus 50 may be used for interpolation processing or the like. For example, the image pickup device 2 detects the first depth information by the distance measuring unit 16, calculates the second depth information based on the image captured by the image pickup unit 15 and the image received from the image pickup device 50, and obtains the second depth information. The first depth information may be interpolated using the two depth information.

The communication unit 55 of the image pickup apparatus 50 can communicate with the communication unit 30 of the information processing apparatus 3 by wire or wirelessly. The communication unit 55 transmits, for example, the setting information of the rendering process (eg, the viewpoint information of the estimated image) to the communication unit 30 of the information processing device 3. The rendering processing unit 32 of the information processing device 3 performs rendering processing using, for example, model information received from the communication unit 13 of the imaging device 2 and rendering processing setting information received from the communication unit 55 of the imaging device 30. .. The communication unit 30 of the information processing device 30 transmits, for example, the data of the estimated image generated by the rendering process to the communication unit 55 of the image pickup device 50. The display unit 54 of the image pickup apparatus 50 displays the estimated image based on, for example, the data of the estimated image from the information processing apparatus 30.

The communication unit 55 of the image pickup apparatus 50 does not have to communicate with the communication unit 30 of the information processing apparatus 3. For example, the communication unit 55 may transmit the setting information of the rendering process to the communication unit 13 of the image pickup apparatus 2. The communication unit 13 may transmit the setting information received from the communication unit 55 of the image pickup device 50 to the communication unit 30 of the information processing device 3. Further, the communication unit 13 may receive data of an estimated image showing the result of the rendering process from the communication unit 30 of the information processing device 3. The communication unit 13 may transmit the data of the estimated image received from the communication unit 30 of the information processing device 3 to the communication unit 55 of the image pickup device 50. The display unit 54 of the image pickup device 50 may display the estimated image based on the data of the estimated image from the image pickup device 2.

The imaging system 1 or imaging device 2 of each of the above-described embodiments may be, for example, at least a part of a production system or a production support system for video content including CG, or a visualization imaging system for blood vessels and biological tissues. It may be at least part of a biomicroscope system, defect detection system or object measurement system. Further, the imaging system 1 or the imaging device 2 may be, for example, at least a part of a detection system that detects the invasion of an object or an animal into the visual field region or the movement of the object or the animal in the visual field region. This detection system may be, for example, a security system, a nursing care system, a monitoring system, a traffic light system, an information acquisition system for acquiring statistical data such as traffic volume, and the like. Further, the imaging system 1 or the imaging device 2 may be at least a part of the simulator. This simulator may generate, for example, an estimated image when an object (eg, furniture) and an animal (eg, a person, a pet) are placed in a space (eg, a room) modeled by the imaging device 2. In addition, this simulator may generate an estimated image when an object or animal modeled by the imaging device 2 is arranged in a predetermined space (eg, a room).

Further, according to each of the above-described embodiments, the imaging system in the present embodiment includes a first imaging unit that images an object and a first ranging unit that detects a distance from each point on the surface of the object. The first model information including at least one of the shape information and the texture information of the object is calculated using the detection results of the first detection unit and the first detection unit, and the first feature point data is calculated using the first model information. A first imaging device (2, 2a) including a first information calculation unit for generating data, a second imaging unit for imaging an object, and a second measurement for detecting a distance from each point on the surface of the object. The second detection unit including the distance portion and the second detection unit include the detection results of the second detection unit to calculate the second model information including at least one of the shape information and the texture information of the object, and the second model information is used to calculate the second model information. The first feature point data obtained by data communication between the second imaging device (2, 2b) including the second information calculation unit for generating the two feature point data, and the first imaging device and the second imaging device. The configuration may also include a model integration unit (35, 45) that integrates the first model information and the second model information using the second feature point data to generate integrated model information of the object. .. In the case of such a configuration, the model integration unit may be provided in at least one of the first imaging device, the second imaging device, or the information processing device, or is provided in another external terminal (model integration processing device). May be done.

The technical scope of the present invention is not limited to the above-described embodiment or modification. For example, one or more of the requirements described in the embodiments or modifications described above may be omitted. In addition, the requirements described in the above embodiments or modifications can be combined as appropriate.

The number of imaging devices included in the imaging system 1 may be three or more. For example, the imaging system 1 may include four imaging devices, two of which include a model integration unit and the remaining two units which do not have a model integration unit. For example, the image pickup apparatus provided with the model integration unit may receive model information from the image pickup apparatus not provided with the model integration unit to perform model integration. In this case, the first integrated model information and the second integrated model information are calculated, and the information processing apparatus 3 may integrate the first integrated model information and the second integrated model information. The imaging device according to the present embodiment may be a wearable terminal, and the main body thereof is provided with a mounting portion that can be worn on an arm or head, and an input portion that can perform voice input, input buttons, and touch operations. You may be.

1 ... Imaging system, 2 ... Imaging device, 3 ... Information processing device, 11 ... Detection unit, 12 ... Information calculation unit, 13 ... Communication unit, 15 ... Imaging unit , 16 ... Distance measuring unit, 18 ... Image sensor, 19 ... Irradiation unit, 21 ... Image sensor,
35 ... Model integration department, OB ... Object

Claims (1)

With the main body
An imaging unit that captures an image of an object, a detection unit that includes a distance measuring unit that detects a distance from each point on the surface of the object, and a detection unit.
An information calculation unit provided in the main body unit and using the detection result of the detection unit to calculate at least one of the shape information and the texture information of the object.
An imaging device including a communication unit that transmits a calculation result of the information calculation unit.

Images