JP2008102961A - Image processing system and method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing system processing various photographed images according to user's demands by synthesizing simulation images corresponding to the photographed images with digital image data. <P>SOLUTION: The image processing system in moving images comprised of frames of a plurality of photographed images comprise: an acquisition means for acquiring photography information regarding the photographed images with the photographed images obtained by photographing a subject; an identification means for specifying and identifying the subject in the photographed images using the photography information; a database which stores three-dimensional structure data of the subject; a reading means for reading the three-dimensional structure data of the subject identified by the identification means from the database; an image creation means for creating simulation images of the identified subject based on the read three-dimensional structure data of the subject and an editing means for editing the moving images using the simulation images and the photographed images. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing system and method, and more particularly, an image obtained by photoelectrically reading an image of a subject photographed on a photographic film by a conventional camera to obtain digital image data, or an image photographed by a digital camera. The present invention relates to image processing when the digital image data is acquired and subjected to predetermined image processing and then output to a photographic print or display device reproduced as a visible image.

  Currently, printing of images taken on photographic films (hereinafter referred to as films) such as negative films and reversal films onto photosensitive materials (printing paper) is performed by projecting the film images onto photosensitive materials and exposing the photosensitive materials to surface exposure. The so-called direct exposure (analog exposure) is performed.

  On the other hand, in recent years, an image of a shooting frame photographed by a printing apparatus using digital exposure, that is, a conventional camera and recorded on a photographic film is photoelectrically read, and the read image is obtained as digital image data. Later, or after obtaining digital image data of an image of a shot frame taken by a digital camera, various image processing is performed to obtain image data for recording, and the photosensitive material is scanned with recording light modulated in accordance with the image data. Digital photo printers that have been exposed to record images (latent images) to produce (finished) photographic prints have been put into practical use.

In digital photo printers, images can be converted into digital image data, and the exposure conditions during printing can be determined by image data processing. Therefore, correction of image skipping and blurring caused by backlighting, flash photography, etc., and sharpness (sharpening) Processing, color feria and density feria correction, underexposure and overexposure correction, peripheral light insufficiency correction, and the like can be suitably performed to obtain a high-quality photographic print that cannot be obtained by conventional direct exposure.
In addition, composition of a plurality of images, image division, and composition of characters can be performed by image data processing, and a photo print that is freely edited / processed according to the application can be output.
Furthermore, according to the digital photo printer, not only can an image be output as a photographic print, but also image data can be supplied to a computer or the like, or stored in an image recording medium such as a floppy (registered trademark) disk. The image data can be used for various purposes other than photography.

  Such a digital photo printer basically includes a scanner (image reading device) that photoelectrically reads an image recorded on a film, and an image processing device that processes the read image to record image data (exposure conditions). And a printer (image recording apparatus) that scans and exposes a photosensitive material in accordance with the recorded (output) image data, develops it, and produces a photographic print.

In a scanner, reading light emitted from a light source is incident on a film to obtain projection light carrying an image photographed on the film, and this projection light is imaged on an image sensor such as a CCD sensor by an imaging lens. Then, the image is read by photoelectric conversion, subjected to various image processing as required, and then sent to the image processing apparatus as film input image data (image data signal). The image processing device sets image processing conditions from input image data read by a scanner or read from an image recording medium and sent to the image processing device, and inputs image processing according to the set conditions. It is applied to the image data and sent to the printer as output image data (exposure conditions) for image recording.
In a printer, for example, if the apparatus uses light beam scanning exposure, the light beam is modulated in accordance with output image data sent from the image processing apparatus, and the photosensitive material is scanned and exposed (baked) two-dimensionally. Then, a latent image is formed, and then a predetermined development process or the like is performed to output a photographic print in which an image photographed on the film is reproduced as a visible reproduction image.

  In addition to a method of reading a photographed image with a scanner and obtaining digital image data, for example, a digital image photographed with a digital camera or the like and recorded on a floppy (registered trademark) disk or an image recording medium such as MO or Zip. Image data can be obtained, and digital image data can be obtained via a network to be input image data. Image processing conditions are set from the input image data, and image processing according to the set conditions is performed. And output to the printer as output image data (exposure conditions) for image recording, and can be printed out as a visible reproduction image.

In such a digital camera, there is a limit to the resolution of an image sensor such as a CCD, so there is a limit in enlarging an image captured by these cameras, and in the case of a silver halide photographic camera, the graininess of the film for shooting is limited. There was a limit.
In view of this, a technique has been proposed in which a plurality of image data shot in the high-definition shooting mode is combined based on shooting control information so that a high-definition print can be obtained without increasing the size of the camera ( For example, see Patent Document 1.)

  In addition, according to the digital photo printer, not only the image photographed on the film by the conventional camera or the digital image photographed by the digital camera is output as a photographic print, but the image can be captured by inappropriate photographing conditions at the time of photographing. Even those with poor reproducibility can be digitally imaged to correct backlit scenes, high-contrast scenes, underexposure negatives, sharpness, lack of ambient light, etc., and easily print high-quality and beautiful images. As a feature that can be output as

In the current photo print service by digital processing of digital photo print, when the shooting date and time is optically recorded on the film, or in the case of the new photo system APS in which the shooting date and time is magnetically recorded, it is shot on the finished print. You can receive a service that displays the date and shooting time.
The service is useful for looking back at the state of shooting by looking at the printed image and the shooting date and time.
However, there are cases where it is difficult to sufficiently look back on various situations at the time of shooting only from information on the shooting date and shooting time. In other words, when viewing images of these photographic prints at a later date, the printed images are looked back at the shooting date and time, but it is often impossible to fully remember what the purpose was and how the image was taken under the circumstances.
Further, for example, when a mountain is photographed in the background of an image of a person, there are cases where it is desired to know the name, altitude, and history of the mountain in the background.
On the other hand, in addition to the shooting date and shooting time, it is often possible to easily look back at the time of shooting if there is incidental information, shooting location, and incidental information related thereto.

  On the other hand, with the spread of personal computers in recent years, it has become easier to perform digital image processing with personal computers, and it is also possible for individuals to combine images by using a scanner or digital camera, or to modify images using photo retouching software. Digital images including photographic prints by digital processing are highly entertaining, in which images are changed according to preference by image composition and image correction.

Therefore, the digital image data photoelectrically taken from the image recorded on the film and the digital image data obtained by photographing with a digital camera or the like are subjected to predetermined image processing to obtain output image data, and this output When outputting a visible playback image according to the image data as a photo print or displaying it on a display device such as a monitor, the shooting status and shooting of the shot image based on the shooting date and time, shooting position, etc. according to the user's wish Additional information associated with the captured image, such as additional information related to the subject of the image, can be added to the photo print or display image, allowing the user to easily look back at the time of shooting by looking at the visible playback image, and to print or display the photo. Techniques have been proposed for enhancing the functions of images and providing users with photo prints and display images that are rich in entertainment according to user preferences. For example, see Patent Document 2.).
JP-A-10-136298 JP 2000-66312 A

However, the invention described in Patent Document 1 requires a plurality of times of shooting to create an image, which is troublesome and requires a special shooting mode function for the camera.
In addition, the invention described in Patent Document 2 provides an image in which a subject cannot be seen because it is hidden in a cloud or a subject is out of focus when providing a photo print or a display image that is rich in entertainment according to user's preference. In addition, a CG image is created, and the CG image and the actually captured image are pattern-matched or synthesized. Therefore, the output image is not made to have high image quality.
The present invention has been made in view of such problems, and digital image data photoelectrically read from an image recorded on a film, or digital image data obtained by photographing with a digital camera or the like, It is an object of the present invention to provide an image processing system capable of processing various photographed images according to a user's desire by synthesizing simulation images corresponding to the photographed images.

In order to solve the above-mentioned problem, the present invention specifies and identifies a subject in the photographed image using an acquisition unit that obtains photographing information regarding the photographed image together with a photographed image obtained by photographing the subject. Identification means for performing, image creation means for creating a simulation image corresponding to the photographed image based on the identified three-dimensional structure data of the subject or the two-dimensional data of the subject, and photographing based on the simulation image Provided is an image processing system comprising processing means for correcting a defective portion of an image or correcting the density and color of a photographed image.
Further, the processing means compares the captured image with the simulation image, extracts a shadow area in the captured image, corrects the density by increasing the luminance of the shadow area in the simulation image, and is corrected. The simulation image may be combined with the captured image.
In order to solve the above-described problem, the present invention specifies an object in the captured image by using an acquired unit that acquires captured information related to the captured image together with a captured image of the captured object, and the captured information. Identifying means for identifying, a database storing the three-dimensional structure data of the subject, a reading means for reading out the three-dimensional structure data of the subject identified by the identifying means from the database, and the three-dimensional structure of the read subject Based on the data, image creating means for creating a simulation image of the subject corresponding to the photographed image, and correcting a defective portion of the subject in the photographed image based on the simulation image of the subject, or of the photographed image Processing means for correcting the density and color of the subject, and the image creating means Based on the structure data, a simulation image of the subject and the shadow area of the subject is created, the processing means compares the captured image with the simulation image, extracts a shadow area in the captured image, and performs the simulation An image processing system is provided, wherein density is corrected so as to increase luminance of the shadow area in an image, and the corrected simulation image is synthesized with the captured image.
Further, the processing means compares the captured image with the simulation image, extracts a high brightness area in the captured image, corrects the density so as to reduce the brightness of the high brightness area in the simulation image, The corrected simulation image may be combined with the captured image.
In addition, the reading unit reads a shooting condition based on the shooting information and a condition input by a user, and the image creating unit reflects a mean light state according to the read shooting condition. The image may be created, and the processing unit may correct the color of the captured image in accordance with the color of the simulation image, and synthesize the corrected captured image and the simulation image.
The image creating means creates a simulation image at a predetermined resolution which is higher than that of the photographed image, and the processing means compares the photographed image with the simulation image, and compares the density of the simulation image and At least one of the colors may be corrected according to the captured image, and the corrected simulation image may be combined with the captured image.
Further, the image creation means creates the simulation image at a resolution higher than the resolution of the photographed image, and the processing means creates a magnified image by enlarging the photographed image, and the magnification ratio of the photographed image is A ratio is determined based on the enlargement ratio from a ratio that is set so that the ratio of the simulation image increases as it increases, and based on the determined ratio, the pixel value of the enlarged image and the pixel value of the simulation image And the enlarged image and the simulation image may be combined.
Further, the image creation means sets at least one of size, sharpness, and contrast separately for the background portion and the main subject in the captured image, and individually simulates the simulation image of the background portion and the main subject. Images and images may be created.
Further, the image creating means may create a simulation image including a region that is not photographed in the photographed image, the field of view being enlarged around the main subject of the photographed image.
Further, the image creating means creates only the background of the photographed image as a simulation image, and the processing means compares the photographed image with the simulation image, and at least one of density, color, blur condition, and camera exposure. One difference may be detected, the simulation image may be corrected according to the difference, and the corrected simulation image and the captured image may be combined.
The image creating means creates a simulation image of the background of the photographed image, and the processing means corrects the simulation image based on the photographed image, and synthesizes the corrected simulation image with the background of the photographed image. May be.

In addition, the present invention is an image processing system for a moving image including a plurality of frames of captured images, and using the captured information, an acquisition unit that acquires captured information about the captured image together with a captured image of the subject. Identifying means for identifying and identifying a subject in the photographed image, a database storing the three-dimensional structure data of the subject, and a reading means for reading out the three-dimensional structure data of the subject identified by the identifying means from the database And an image creating means for generating a simulation image of the identified subject based on the read three-dimensional structure data of the subject, and an editing means for editing a moving image using the simulation image and the captured image. An image processing system is provided.
The image creating means creates a simulation image of the subject corresponding to the photographed image between the frames of the respective photographed images, and the editing means uses the simulation image as the photographed image between the frames. It may be inserted.
The image creating unit may create simulation images corresponding to the captured images before and after the moving image that has not been captured, and the editing unit may insert the simulation image before and after the moving image.
The editing means may edit a moving image by correcting a specific subject in the simulation image, replacing the corrected simulation image with the captured image.
The editing unit may correct a background near the specific subject in the simulation image.
Further, the database may include communication means, and the stored three-dimensional structure data of the subject may be updated based on information provided from outside via the communication means.
The present invention is also an image processing method for a moving image including a plurality of frames of captured images, the acquisition step of acquiring shooting information about the captured image together with the captured image of the subject, and using the shooting information. An identification step for identifying and identifying a subject in the captured image, a reading step for reading out the three-dimensional structure data of the identified subject from a database, and generating a simulation image based on the read-out three-dimensional structure data of the subject An image processing method comprising: an image creation step; and an editing step for editing a moving image using the simulation image and the captured image.
Here, the image creating step creates a simulation image of the subject corresponding to a photographed image between the frames of the respective photographed images, and the editing step includes the simulation image between the frames and the frames. You may insert as a picked-up image.
Further, the image creation step may create simulation images corresponding to the captured images before and after the unmoved moving image, and the editing step may insert the simulation image before and after the moving image.
Further, the editing step may correct a specific subject in the simulation image, replace the corrected simulation image with the captured image, and edit the moving image.
The editing step may correct a background near the specific subject in the simulation image.

  According to the present invention, various photographed images can be added to digital image data photoelectrically read from an image recorded on a film or digital image data obtained by photographing with a digital camera or the like according to a user's request. It is possible to provide an image processing system capable of performing the above processing.

  An image processing system according to the present invention will be described below in detail based on a preferred embodiment shown in the accompanying drawings. First, an image processing system according to the present invention will be described with reference to FIGS.

FIG. 1 is a block diagram showing an outline of a digital photo printer constituting the image processing system according to the first embodiment of the present invention.
A digital photo printer 10 shown in FIG. 1 basically includes a scanner (image input means) 12 that photoelectrically reads an image photographed on a film F, and image data for output by processing the read image data. In addition, the image processing device 14 that performs operation and control of the entire digital photo printer 10 and the light exposure material (printing paper) that has been modulated in accordance with the image data output from the image processing device 14 are subjected to image exposure. And a printer 16 (image output means) for performing development processing (finished) and outputting as a print.

  The image processing apparatus 14 is read by the scanner 12 and an operation system 18 having a keyboard 18a and a mouse 18b for inputting various conditions input and settings, process selection and instructions, color / density correction instructions, and the like. Connected to a display (image display device) 20 for displaying images, various operation instructions, setting / registration screens for various conditions, and the like.

  The scanner 12 is a device that photoelectrically reads an image photographed on the film F, and includes a light source 22, a variable aperture 24, a diffusion box 28 that makes reading light incident on the film F uniform in the surface direction of the film F, The image forming lens unit 32, a CCD sensor 34 that is a photo sensor that reads a photographed image of the film, an amplifier 36, and an A / D (analog / digital) converter 38, It is composed of a dedicated carrier 30 that can be attached to the main body.

For the carrier 30, for example, a special carrier corresponding to a long film such as a 24-size 135-size film or a new photographic system film (APS cartridge) is prepared, and the film F is placed at a predetermined reading position. While the image sensor is being held, the longitudinal direction of the film F is conveyed in the sub-scanning direction perpendicular to the extending direction (main scanning direction) of the line CCD sensor of the CCD sensor 34, and the reading position is sandwiched in the sub-scanning direction. A pair of transport rollers 30a and 30b, a mask 29 having a slit 29a extending in the main scanning direction, which corresponds to the reading position, restricts the projection light of the film F into a predetermined slit shape, and further magnetically And a reading / writing device 31.
The film F is held by the carrier 30 and conveyed in the sub-scanning direction, and reading light is incident on the film F. Accordingly, the film F is two-dimensionally slit-scanned by the slits 29a extending in the main scanning direction, and the images of the frames shot on the film F are read.

The CCD sensor 34 includes a line CCD sensor 34R that reads an R image, a line CCD sensor 34G that reads a G image, and a line CCD sensor 34B that reads a B image. The line sensor is in the main scanning direction. It is extended.
The projection light of the film F is separated into the three primary colors R, G and B by this CCD sensor and is read photoelectrically. Reading light emitted from the light source 22, adjusted in light quantity by the variable aperture 24 and made uniform through the diffusion box 28 is incident on the film F being conveyed while being held at a predetermined reading position by the carrier 30 and is transmitted therethrough. The projection light carrying the image photographed on the film F is obtained.
The projection light of the film F is imaged on the light receiving surface of the CCD sensor 34 by the imaging lens unit 32, is read photoelectrically by the CCD sensor 34, and the output signal is amplified by the amplifier 36 and A / D converted. The digital image data is converted by the device 38 and sent to the image processing apparatus 14 as input image data.

The scanner 12 constituting the digital photo printer 10 is not limited to the one using the above-described slit scanning, and may be a CCD area sensor using surface exposure that reads the entire surface of one frame image at a time.
In that case, R, G, and B color filters are provided between the variable aperture 24 and the diffusion box 28 shown in FIG. Projection light carrying an image of this frame photographed on the film F may be obtained by entering and transmitting one frame. In this case, it is necessary to sequentially perform the color filter for R, G, and B three times.

Further, reading of an image by the CCD sensor in the scanner 12 is performed by performing a pre-scan for reading an image at a low resolution in order to determine image processing conditions and the like prior to a main scan for reading an image to output a photographic print. After the processing conditions are determined and the operator (or user) confirms and adjusts on the display 20, a main scan for reading an image at a high resolution is performed. Therefore, the scan is performed twice: a pre-scan and a main scan.
Therefore, when a CCD area sensor using surface exposure is provided with R, G, and B color filters, it is necessary to scan three times using the R, G, and B color filters. A scan will be performed.
In the case of using a line CCD sensor, since only two times are required, it is advantageous for rapid processing. In pre-scanning, all images on film F are captured by pre-scanning at once, and after setting the image processing conditions, main scanning is performed. However, pre-scanning and main scanning are sequentially performed on film F frame by frame. You may go.

In the present invention, the scanner 12 is not limited to the one using such slit scanning, and may use a surface exposure that reads the entire image of one frame at a time.
In this case, for example, an area CCD sensor is used, and each color filter of R, G, and B is sequentially inserted between the light source and the film F, and an image taken on the film is read by the area CCD sensor. May be separated into three primary colors and sequentially read.

As described above, the output signal (image data) from the scanner 12 is output to the image processing device 14.
The image processing device 14 reads image data, subject information, and the like from an image data recording medium such as smart media for recording digital image data obtained by photographing with a digital camera or the like, or FD, CD- A drive device 42 for outputting processed image data is connected to a general-purpose image data recording medium such as R, MO, DVD or Zip.

  Furthermore, the image processing apparatus 14 according to the present embodiment is directly connected to various image data supply sources such as a personal computer and a digital camera, and devices that collectively manage attached information of each subject person via a cable. Alternatively, a slot 44 for acquiring digital image data, shooting information thereof, attached information of a subject, and the like are also connected through a communication network.

Therefore, in the present invention, in addition to the scanner 12 that photoelectrically reads an image photographed on a film such as a negative or a reversal, an image reading apparatus that reads an image of a reflected original, or communication means such as computer communication (something via a modem) Imaging devices such as digital cameras and digital video cameras, built-in memory, image recording media for digital cameras such as PC cards and smart media, FD (floppy (registered trademark) disk), MO (magneto-optical recording media), etc. Various image data supply sources such as general-purpose image recording media can be used.
Furthermore, these can be connected to the image processing device 14 directly or via the driving device, and the image processing device 14 can receive digital image data, its photographing information, and subject information from these image data supply sources. it can.

  In the illustrated example, input signals (digital image data, photographing information, additional information) are input to the image processing device 14 from various image data supply sources such as the scanner 12 and the drive device 42. A case where digital image data or the like is mainly supplied from the scanner 12 to the image processing apparatus 14 will be described as a representative example.

FIG. 2 is a block diagram showing a schematic configuration of the image processing apparatus 14 according to the present invention.
The image processing device 14 performs predetermined image processing on the image data (input image) read by the scanner 12 and sent to the image processing device 14 as digital data, and outputs the image data to the printer 16 or the display 20. , Mainly composed of a data processing unit 38, a pre-scan memory 40, a main scan memory 42, a pre-scan image processing unit 44, a main scan image processing unit 46, a condition setting unit 48, and a subject information acquisition means 62.
In the present embodiment, the image processing apparatus 14 is also acquired from a subject information acquisition unit 62 that acquires information about a subject person from an input image, acquired by a camera at the time of shooting, or acquired from a device that collectively manages information. The auxiliary information is arranged in a subject identification unit 48a, a CG image creation unit 48b, and an image processing unit 54 for identifying a subject based on the consistency between subject information such as a person or landscape and the subject information. And image synthesizing means 48c for synthesizing images in association with the subject person in the corresponding photographed image.

In the data processing unit 38, R, G, B digital image data (input image data signal) output from the scanner 12 is subjected to log conversion, DC offset correction, dark correction, shading correction, and the like. The can (image) data is stored (stored) in the pre-scan memory 40, and the main scan (image) data is also stored (stored) in the main scan memory 42, respectively.
Each image data stored in the pre-scan memory 40 and the main scan memory 42 is called by the pre-scan image processing unit 44 and the main scan image processing unit 46, respectively, in order to perform image processing and output as necessary.

The prescan image processing unit 44 includes a prescan image processing unit 50 and a data conversion unit 52. The prescan image processing unit 50 includes color balance adjustment, contrast adjustment, contrast correction, brightness correction, sharpness processing and covering. Other image processing such as baking processing is performed.
In addition, the data conversion unit 52 converts the image data that has been subjected to the image processing by the pre-scan image processing unit 50 into image data corresponding to display on the display 20 using a 3DLUT or the like.

The main scan image processing unit 46 includes an image processing unit 54 and a data conversion unit 66. The main scan image processing unit 62 performs various types of image processing on the main scan data according to the image processing conditions set by the condition setting unit 56 as described below, for example, color balance adjustment, contrast correction (gradation processing), Processing such as brightness correction, saturation correction, sharpness processing, and dodging processing is performed.
The data conversion unit 58 converts the image data after the image processing into a format corresponding to each image output means such as the printer 16, the drive device 42, and the slot 44.

  The subject information acquisition unit 62 includes a subject information selection unit 62a and a database 62b. The subject information acquisition unit 62 specifies a subject of an image captured using subject information obtained from the scanner 12 and map information in the database 62b. Data relating to the identified subject is sent to the subject identifying means 48a in the condition setting unit 48.

As a method for specifying the subject, in the case of the new photographic system APS, a magnetic recording layer is formed on the film F corresponding to the new photographic system APS, and photographing information can be recorded. When taking a picture with a new APS compatible camera using the Positioning System and adding a direction indicator for a direction finder, the taken latitude, longitude and altitude, as well as the shooting azimuth for the horizontal and vertical planes are recorded. In addition, the shooting magnification at the time of shooting can also be recorded.
The recorded information is read by a magnetic reading / writing device 31 provided on the carrier 30 of the scanner 12, and sent from the scanner 12 to the subject information selection means 62a as subject information through a different route from the image data. The subject information includes a first data configuration including a shooting position, a second data configuration including a shooting position and a shooting direction, a third data configuration including a shooting position and a shooting magnification, and a shooting position, a shooting direction, and a shooting magnification. The subject information selection unit 62a acquires subject information associated with a captured image.

  The subject information selection means 62a uses the data in any one of the first to fourth data configurations, which are the acquired subject information, including the shooting position, shooting direction, shooting magnification, and the like, in the map 62b in the database 62b. With reference to the information, the subject and shooting point of the shot image are specified.

  The positioning accuracy for the position of longitude, latitude, and altitude obtained using GPS is within 100 m, and there is no practical problem. With regard to the azimuth angle, the azimuth can be accurately measured by using the direction indicator for the direction detector. .

In this way, the subject information selection unit 62a can specify the subject to be photographed and the photographing point as an object on the map information.
The shooting point may include not only the position of the subject in the shot image, that is, the shooting point (position), but also a shooting position such as the position of the photographer or the camera, that is, the shooting position. Information relating to the shooting position can be used in moving image shooting, for example, when a moving state corresponding to an intermediate position of a subject is estimated based on the frame of the shot moving image and an attempt is made to create a CG image.

  In addition, the method by which the subject information selection unit 62a identifies the subject of the photographed image is not limited to this, and by using wireless communication such as an IC tag (see, for example, JP-A-2002-305714), Subject information may be acquired via the slot 44 to specify an image subject.

The condition setting unit 48 includes a subject identification unit 48a, a CG image creation unit 48b, and an image synthesis unit 48c. The subject identification unit 48a identifies a photographed subject, creates a CG image using the CG image creation unit 48b, and captures the image. The corrected CG image is synthesized with the image. The composite image is sent to an output device such as the printer 16 or the display 20 and is printed by the printer 16 or displayed on the display 20.
Note that various corrections can be applied to the CG image before synthesis in accordance with the user's request. Such corrections include correction of defective portions of the captured image, correction of the density and color of the captured image, adjustment of the luminance according to the captured area, pseudo-high resolution compared to the captured image, Correction for enlarging the image, or correction for emphasizing a specific subject in the captured image.

The subject identifying unit 48a identifies the subject in the photographed image by acquiring and collating the photographed image processed by the pre-scan image processing unit 44 with the photographed subject specified by the subject information selecting unit 62a. .
The subject identification unit 48a receives prescan data from the prescan image processing unit 50 in the prescan image processing unit 44, and uses subject information such as information for subject identification and subject position information from the subject information selection unit 62a. get. For example, a face image of a subject person is preferably exemplified as information used for identifying a subject that is a person. In this case, the subject identification unit 48a extracts the face image of the subject person in the image from the prescan data.
At this time, the subject information of the subject person includes face image data, and the subject identifying unit 48a performs person identification by pattern matching the extracted face image and the face image in the subject information.

In the CG image creation means 48b, a CG image is created as a simulation image based on the subject information obtained through the subject identification means 48a and the three-dimensional structure data of the specific subject, and a predetermined CG image is added to the CG image according to a user instruction. Correction is applied.
The three-dimensional structure data relating to a structure such as a building is acquired by, for example, laser measurement from a helicopter and aerial photography using an HD (High Definition) camera and a high-definition camera. The three-dimensional structure data includes DEM (Digital Elevation Map) indicating the height information of a certain area acquired by laser measurement, and aerial photographs of buildings acquired by aerial photography with a high-definition camera. .
Such three-dimensional structure data is stored in the database 62b, and the CG image creation means 48b acquires the three-dimensional structure data of the specific subject from the database 62b via the subject information selection means 62a, and obtains the three-dimensional structure data. Based on the above, a CG image of a specific subject is created by a known CG image creation technique. Examples of such a CG image created include natural structures, artificial structures, and human bodies.

As for the three-dimensional structure data related to the article, the three-dimensional structure data can be obtained from the three-dimensional structure data stored in a storage medium such as an IC tag attached to the article.
Further, the data used for creating the CG image is not limited to the three-dimensional structure data, and may be two-dimensional image data. Examples of CG images obtained from such data include clouds, water surfaces, and trees.

FIG. 3 is a block diagram showing a schematic configuration of the image synthesizing means, and shows processing for synthesizing CG image data with captured image data to generate a synthesized image.
The image synthesizing unit 48c includes a smoothing filter, a high-frequency component processing unit, and a low-frequency component processing unit. The captured image data obtained from the subject identifying unit 48a and the CG image data obtained from the CG image creating unit 48b. And the combined image is sent to the output devices 16 and 20.

In the image synthesizing unit 48c, the captured image obtained from the subject identifying unit 48a is branched, and the low frequency component of the captured image is obtained by passing one captured image through the smoothing filter 49a, and the low frequency component is obtained from the other captured image. A high frequency component can be obtained by removing.
In this way, the captured image is separated into the image low frequency component and the image high frequency component and sent to the image low frequency processing means 51b and the image high frequency processing means 51a, and the correction processing described below is performed.
On the other hand, the CG image obtained from the CG image creating means 48b is also branched in the same manner, and a low frequency component is obtained by passing one CG image through the smoothing filter 49b, and a low frequency component is removed from the other CG image. A high frequency component is obtained, and the CG image is separated into an image low frequency component and an image high frequency component and sent to the image low frequency processing means 51b and the image high frequency processing means 51a, and correction processing described below is performed.
Further, as information obtained from the CG image creation means 48b, there is area information other than the CG image, and the area information is information for specifying an area (a shadow area or a high luminance area) to be corrected, and a high frequency component process It is sent to the means 51a and the low frequency processing means 51b.

  The low frequency component processing means 51b acquires the low frequency component of the captured image and area information such as a shadow area or a high luminance area, and performs a predetermined correction on the low frequency component of the predetermined image area, The image high-frequency component processing means 51a acquires information on the high-frequency component of the photographed image, performs a predetermined correction on the high-frequency component in a predetermined image area, and synthesizes both image components so that the user can Generates a composite image with the desired correction. The composite image is output as an output image by image output means such as the printer 16 or the display 20.

As shown in FIG. 4A, the density of the shadow area 41 of the photographed image is high, and a composite image of the shadow area 41 having the gradation density like the CG image as shown in FIG. 4B is desired. For example, the low frequency component processing means 51b determines an area (shadow area) to be corrected from the image area information, and the density gradation of the low frequency component of the photographed image is determined for the area. The composite image desired by the user can be generated by performing correction according to the components and adding both image components without correcting the high frequency components by the high frequency component processing means.
Further, the high frequency component processing unit determines an area to be corrected from the image area information (for example, an area where the ratio of the high frequency component between the captured image and the CG image has a difference of a predetermined value or more), and the high frequency component of the captured image is determined. If the amplitude is adjusted to the high frequency component of the CG image, and the low frequency component processing means does not correct the low frequency component but adds the image component, the sharpness can be corrected in addition to the gradation density. it can.
Note that the composite image is not limited to the correction of only the shadow area or the high-luminance area of the captured image, and both areas can be corrected simultaneously, and a composite image in which various corrections are performed on the captured image can be generated.

In the above embodiment, image area information is extracted and a CG image is generated according to the image area. However, a CG image may be generated in consideration of an average light beam state. The average light condition refers to the condition of light that varies depending on the shooting conditions such as season, time of day, or weather, and refers to the shooting information, obtains information on shooting conditions from the database, and includes information on the shooting conditions. Based on this, a CG image is created.
By correcting the captured image to generate a composite image so as to match the color of the CG image generated in this way, a composite image in which the user has made corrections desired can be generated. .
For example, if the photographed image was taken on a trip and the weather at that time was raining, a clear background at the photographing location was created as a CG image and photographed according to the background of the CG image. The image can be corrected. Such a CG image is generated by the CG image creating means 48b based on the two-dimensional data acquired via the database 62b or the slot 44.

Alternatively, a CG image may be created with a predetermined resolution higher than that of the captured image, and the density and color of the CG image may be corrected to match the captured image. By doing so, it is possible to correct the captured image so that the density and the color can be harmonized with higher resolution.
Alternatively, the captured image may be enlarged by a known method at an arbitrary ratio, and a composite image corrected by referring to a CG image having a similar ratio may be generated.
Further, when the captured image and the CG image are combined, the pixel value C of the combined image is combined using the pixel value A of the captured image and the pixel value B of the CG image as shown in the following formula (1). Also good. Further, it is preferable that the weighting coefficient K is controlled so as to increase as the enlargement ratio increases.

(Equation 1)
C = A × (1-K) + B × K (1)
K: Weighting coefficient K = 0.0 to 1.0

  In general, when a captured image is enlarged, the image becomes blurred. However, with the above-described configuration, as the magnification ratio of the captured image increases, the ratio of CG in the composite image gradually increases. Compared with this, it is possible to create a clearer and more detailed enlarged image.

Further, the background portion of the photographed image and the CG image of the main subject may be created separately. That is, when there are a plurality of main subjects in the captured image and it is desired to emphasize a specific subject, a CG image is created in which only the specific subject is captured at a short distance.
For example, when the captured image is reduced like a thumbnail image, the main subject is also reduced. However, by reducing the reduction rate of the main subject and performing size enhancement, it becomes easier to grasp the contents of the image than in the past. . Also, if the background image feels smaller than the impression in the captured image, the background image can be enlarged.
Thus, by separately creating the background portion of the captured image and the CG image of the main subject, it is possible to individually enhance sharpness and contrast, not limited to size.

  Also, a CG image with an enlarged field of view may be created around the main subject of the captured image. Create a CG image of the surrounding background that is not photographed in the photographed image, correct it to match the density and color of the photographed image, and combine it with the photographed image to grasp the entire background centered on the main subject. Can do.

Furthermore, when other people or unnecessary objects are photographed in the photographed image, the background portion of the photographed image may be created as a CG image. Based on the difference in the common part between the CG image and the photographed image, a difference in the camera exposure, blur condition, gradation density or color of the CG image compared with the photographed image is detected, and the CG is detected according to the difference. The image can be corrected.
By combining the CG image with a captured image using a CG image from which obstacles such as other people and unnecessary objects are removed, correction according to the difference from the captured image is performed on the CG image. Can do well.

FIG. 5 is a diagram for explaining image processing in moving image shooting, and shows a state where a user has shot a moving subject.
The solid line represents the subject at a position corresponding to the frame (frame) of the photographed image taken by the user, and the dotted line represents the subject at the position corresponding to the frame to be inserted between the frames of the photographed image. The alternate long and short dash line represents the movement path of the subject.
The moving path, speed, and direction of the subject can be estimated by a known technique using a navigator device or sensors attached to the subject.
In order to estimate the moving path of the subject, it is necessary to recognize the user's camera position, but the user's camera position can be obtained from GPS or other information included in the subject information.

In this embodiment, since an intermediate timing image between frames of a captured image in a moving image is created as a CG image, the state of a subject at an intermediate timing is estimated using a known technique, a CG image is created, and an intermediate timing image is created. Insert between images as an image.
For example, when a user shoots a moving image composed of 30 frames per second, a moving image is generated by inserting an image created with CG images one by one between the frames. The video can be edited to include 60 frames per second.

In addition, when the motion of the subject exceeds a predetermined speed in moving image shooting, the flowed image is displayed. A portion of the subject corresponding to the flowed video is created as a CG image, and the CG image is combined with the portion of the image corresponding to the flowed video, thereby displaying a smooth video with no flow.
Combining the image of the subject created with the CG image with the portion corresponding to the video that has flowed in this way functions as camera shake correction when applied to a still image.
Conversely, an image that flows as a CG image may be created and combined with a captured image to give a sense of speed to the video.

Other Embodiments In FIG. 6, a server (home server) 92 is connected through a network 96. The home server 92 has a database 94. In the present embodiment, the database 94 is shown as a separate body from the home server 92, but may be integrated with the home server 92.
A DB manager connected to the camera 90 owned by the user via the network 96 provides the user with various data as photographing request information. The shooting request information provided by the DB manager includes the position, direction, magnification, and sampled image prediction sample of a specific subject.
For example, when requesting the user to take an image of a mountain imaged from X in the map data shown in FIG. 7, the DB manager provides the user with a sample image that is a predicted sample image as shown in FIG. Send.
A plurality of specific subject data is registered in the database manager database 94. For example, when the specific subject is a mountain, the specific subject data is mountain data taken from a plurality of directions. The registered data is not limited to specific subject data by direction, but may be specific subject data by conditions such as season, time zone, shooting distance, resolution, or weather.

The method for acquiring such specific subject data is not particularly limited, but for example, recruiting a certain member user of the registration system, requesting the member user to acquire data, and transmitting the specific subject data to the supplier The trader can register the specific subject data. In such a case, the member who transmitted the data registered as the specific subject data may be provided with some benefit such as a print fee or a membership fee discount.
Also, specific subject data with high access to specific subject data can be regarded as a popular shooting spot and displayed on an image map or disclosed to member users.

In addition, when acquiring the specific subject data, if the specific subject data is not registered in the database 94, the specific subject data can be transmitted by requesting a member user.
The database manager can enrich the contents of the image database 94 and can create a CG image based on the latest specific subject data when creating the CG image.
In addition, the member user can be more entertaining with respect to shooting and can also return profits.

Another embodiment using the network 96 will be described. In such an embodiment, the camera 90 is always connected to the home server 92 and can communicate in real time.
The camera 90 transmits the recognized image data and subject information (position, orientation, magnification data, etc.) to the home server 92 via the network 96. Note that the camera 90 may transmit image data with reduced resolution and image feature amounts (such as density histogram).
The home server 92 that has received the transmitted image data and subject information determines whether a CG image of the subject recognized by the camera 90 can be created, and transmits the determination result to the camera 90.

When the home server 92 determines that the home server 92 can create a CG image, the camera 90 that has received the determination result automatically performs shooting.
On the other hand, when the home server determines that the CG image cannot be created, the camera 90 displays a message on the display screen of the camera 90 so as to set the multiple shooting mode or the manual shooting mode to the user. Prompt.

  Furthermore, the recommended mode may be determined according to the image data and displayed on the display screen of the camera 90. Examples of this include the fact that the histogram distribution is biased toward the low or high luminance side and that the density range is wide. With auto, there is a risk of overexposure or blackout in low luminance areas. If there is, you can also recommend multiple exposure shooting or increase or decrease the exposure in the manual.

When the camera 90 that has received the determination result from the home server 92 determines that the home server 92 can create a CG image, the camera 90 captures the target scene normally, zooms to the main subject, and focuses on the main subject. It is also possible to take a picture and send the photography data to the home server 92.
The home server 92 that has received the photographic data creates a CG image according to a predetermined setting, and performs processing such as high resolution, wide-angle image adjustment, and focusing for each subject on the photographic data.
On the other hand, if the home server 92 determines that a CG image cannot be created, the camera 90 displays a message to the camera 90 so as to set a special shooting mode such as divided shooting, multiple zoom shooting, or multiple focus. It may be displayed on the screen to prompt the user.

In addition, when the camera 90 that has received the determination result from the home server 92 determines that the home server 92 can create a CG image, it performs shooting as described above and increases the compression ratio at the time of shooting. The photographing data may be recorded under control.
If the compression rate is high, the image quality usually deteriorates, but a high-resolution image can be created by the image processing described above.

In the above-described embodiment, whether or not the home server 92 can create a CG image is determined for a subject recognized by the camera 90. However, the present invention is not limited to this, and may be determined for each subject.
By doing so, shooting data to be recorded when the home server 92 determines that a CG image can be created can be recorded by controlling the compression rate for each area. Therefore, it is possible to reduce the amount of data to be stored as compared with the case where shooting data is stored without controlling the compression rate for each area.

Further, in moving image shooting, the camera 90 transmits video data to the home server 92, and the home server 92 determines whether or not a moving image frame (frame) can be reproduced as a CG image from the video data. Is transmitted to the camera 90.
When the home server 92 determines that a moving image frame can be reproduced as a CG image, the camera 90 records frames by reducing the number of frames per unit time as compared to normal shooting. Since the deleted frame can be reproduced as a CG image by the home server 92, there is no possibility of dropping frames of the moving image.
By doing so, the amount of data recorded in the camera 90 can be reduced as compared to the normal time.

  As described above, the image processing system according to the present invention has been described in detail. However, the present invention is not limited to the above embodiment, and various improvements and modifications are made without departing from the gist of the present invention. Also good.

It is a block diagram which shows the outline of a digital photo print. It is a block diagram which shows schematic structure of an image processing apparatus. It is a block diagram which shows schematic structure of an image synthetic | combination means. It is a figure for demonstrating the process which synthesize | combines CG image data with picked-up image data, and produces | generates a synthesized image. It is a figure for demonstrating the image process in video recording. It is a figure which shows schematic structure of one Embodiment using a network. It is a figure which shows an example of map data. It is a figure which shows an example of a sample image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Digital photo printer 12 Scanner 14 Image processing apparatus 16 Printer 18a Keyboard 18b Mouse 20 Display 22 Light source 28 Diffusion box 29a Slit 30 Carrier 31 Magnetic reading / writing apparatus 32 Imaging lens unit 38 Data processing part 40 Prescan memory 42 Main scan memory 44 Pre-scan image processing unit 46 Main scan image processing unit 56 Condition setting unit 62 Subject information acquisition means

Claims (11)

An image processing system for moving images consisting of a plurality of shot images,
An acquisition means for acquiring shooting information related to the shot image together with a shot image obtained by shooting the subject;
Identifying means for identifying and identifying a subject in the photographed image using the photographing information;
A database storing the three-dimensional structure data of the subject;
Reading means for reading out the three-dimensional structure data of the subject identified by the identification means from the database;
Image generating means for generating a simulation image of the identified subject based on the read three-dimensional structure data of the subject;
An image processing system, comprising: an editing unit that edits a moving image using the simulation image and the captured image. The image creating means creates a simulation image of the subject corresponding to a shot image between frames of each shot image,
The image processing system according to claim 1, wherein the editing unit inserts the simulation image as a captured image between frames. The image creating means creates simulation images corresponding to the captured images before and after the unmoved video,
The image processing system according to claim 1, wherein the editing unit inserts the simulation image before and after a moving image.   The image processing system according to claim 1, wherein the editing unit corrects a specific subject in the simulation image, replaces the corrected simulation image with the captured image, and edits a moving image.   The image processing system according to claim 4, wherein the editing unit corrects a background near the specific subject in the simulation image.   The image according to claim 1, wherein the database includes a communication unit, and updates the stored three-dimensional structure data of the subject based on information provided from outside through the communication unit. Processing system. An image processing method for a moving image composed of a plurality of shot images,
An acquisition step of acquiring shooting information related to the shot image together with a shot image obtained by shooting the subject;
An identifying step for identifying and identifying a subject in the captured image using the captured information;
A step of reading out the three-dimensional structure data of the identified subject from the database;
An image creating step for generating a simulation image based on the read three-dimensional structure data of the subject;
An image processing method comprising: an editing step of editing a moving image using the simulation image and the captured image. The image creation step creates a simulation image of the subject corresponding to the shot image between frames of each shot image,
The image processing method according to claim 7, wherein the editing step inserts the simulation image as the captured image between frames. The image creation step creates a simulation image corresponding to the captured images before and after the unmoved video,
The image processing method according to claim 7, wherein the editing step inserts the simulation image before and after a moving image.   The image processing method according to claim 7, wherein the editing step corrects a specific subject in the simulation image, replaces the corrected simulation image with the captured image, and edits a moving image.   The image processing method according to claim 10, wherein the editing step corrects a background in the vicinity of the specific subject in the simulation image.

Images