JP2003288601A - Imaging apparatus, image processing apparatus, image processing method, and method of image information classification service - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus, image processing apparatus, image processing method and image information classification service which can re trieve and classify with high precision by recording information effective for the retrieval and classification in conjunction with image data thereby effectively conducting data processing such as retrieval and browsing. <P>SOLUTION: When a target image is analyzed, by using image data taken just before the target image, the background motion of the target image is canceled and areas where the image moves are extracted for image analysis. The obtained information is linked to the image and then recorded. In addition, information on the imaging such as an imaging mode is also recorded together with the image which can be used for image retrieval. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup device and an image processing method capable of searching a picked-up image.

[0002]

2. Description of the Related Art In recent years, with the widespread use of image pickup devices such as digital still cameras, video cameras, infrared cameras, and surveillance cameras, it has been required that the images picked up by these image pickup devices can be searched as needed. ing.
Further, in the search, it is desired that the accuracy is high, the required time is short, and the usability is good. As such a search method, a method may be considered in which information is automatically extracted from the captured image and the extracted information or other information regarding imaging is used for the search. As a method for automatically recognizing a feature of a captured image, Japanese Patent Laid-Open No. 2001-16
No. 7253 is available. According to the present invention, the image pickup apparatus has the person detecting means, and the information on the position of the person in the picked-up image can be extracted using the color information and the shape information. As a method of retrieving an imaged and recorded image, Japanese Patent Laid-Open No. 11-234605 discloses a method of retrieving an imaged image by using imaging information and history information regarding display. In addition, Japanese Patent Laid-Open No. 2001-265794
In the issue, an image is searched based on the stored image data.
An image processing system using an apparatus that performs data processing such as browsing is disclosed.

[0003]

However, in the above conventional technique, for example, in the method of automatically recognizing the features of an image, color information and shape information for extracting a person are obtained from a still image. Therefore, the yield of correct detection is not so high due to the influence of the shooting environment. Also,
Since the entire still image is processed, there is a problem that the processing load is high in processing in the imaging device. In addition, there is a problem that an advanced search cannot be performed with the imaging information and the history information. Further, in the method using the image processing device, all the images are processed when the image data is processed, and it is impossible to preferentially process only some of the images.
Therefore, there is a problem that when a large number of images are processed, it takes a considerable time. An object of the present invention is to perform effective search and classification by recording effective information for search classification in association with an image, and to prioritize image data to prioritize some images. It is to provide an image capturing device, an image processing device, an image processing method, and an image information classification service that can more efficiently perform data processing such as searching and browsing by performing the processing.

[0004]

In order to solve the above-mentioned problems, the invention according to claim 1 is to take an image and to output the image information obtained by the image pickup; While detecting the moving body that has moved in the range, the information of the area occupied by the moving body, the moving direction of the moving body, the number of the moving body of one or more of A moving body detecting unit that obtains and outputs information about a moving body, image information captured by the image capturing unit, and information about the moving body obtained from the image information by the moving body detecting unit in association with each other. An image pickup device comprising: a storage unit that stores information and information about a mobile body in a storage medium.

According to the first aspect of the invention, the area of the moving body obtained from the detection result by the moving body detecting means,
Since the recording means is provided for recording at least one or more of the information such as the moving direction and the number in the storage medium in association with the captured image, it is possible to extract the person, automatically determine the top and bottom of the image, and determine the number of persons. Information (region, moving direction, number, and the like of the moving body) effective for the determination can be added to the captured image at the time of capturing the image.

According to a second aspect of the present invention, in the image pickup device according to the first aspect, the moving body detection means uses a plurality of pieces of image information which are picked up by the image pickup means at different times and have overlapping image pickup ranges. Then, an image portion that moves relatively to the background image portion is detected as a moving body.

According to the second aspect of the present invention, the moving body detecting means uses a plurality of pieces of image information which are picked up by the image pickup means at different times and have overlapping image pickup ranges, and are relative to the background image portion. It is possible to detect a moving image portion as a moving body. That is, since the movement of the background caused by moving the camera can be canceled, the region in which the moving body is moving can be accurately obtained.

The invention according to claim 3 is the invention according to claim 1 or 2.
The image capturing device according to claim 1, further comprising: an estimating unit that estimates whether or not the moving body detected by the moving body detecting unit is a person, and the estimating unit estimates the moving body to be a person. In addition, the storage means is characterized by adding information that the moving body is a person to the information of the moving body stored in the storage medium.

According to the third aspect of the present invention, since it can be estimated whether or not the moving body is a person only for the detected area of the moving body, the processing amount required for this estimation is reduced, and the estimation is performed. Can be done at high speed.

According to a fourth aspect of the present invention, an image pickup means for taking an image and outputting the image information obtained by the image pickup, and a temperature detection for outputting the temperature information obtained by detecting the temperature within the imaged range. Means for estimating whether or not a person is included in the image information from the temperature information or the temperature information and the image information, and when it is estimated that the image information includes a person, the Information of the area occupied by the person, information about the moving direction of the person, information about the number of the person, and a person extraction unit that obtains and outputs information about the person, and an image captured by the image capturing unit. An image characterized by comprising: storage means for associating the image information and the information of the person obtained from the image information by the person extracting means with each other, and storing the image information and the information of the person in a storage medium. It is an imaging device.

According to the invention described in claim 4, since the temperature information is used in the extraction of the information of the person from the image information, the person in the image information can be estimated from the temperature distribution and the human body temperature, for example. Further, since the person can be estimated from the temperature and the skin color or shape in the image information, the person who is moving can be estimated from the temperature and the moving image. Further, even in the case of performing image analysis, since it is sufficient to estimate the human region in advance based on the temperature information and analyze the image of only a limited region, the processing amount for the extraction is reduced, and the extraction can be performed at high speed. Can be done.

According to a fifth aspect of the present invention, an image pickup means for performing image pickup and outputting image information obtained by the image pickup, a parameter selecting means for selecting a parameter relating to the image pickup, and a parameter selected by the parameter selecting means are set. An image pickup device comprising: a recording unit that stores the information indicated in a storage medium in association with the picked-up image.

According to the fifth aspect of the present invention, since there is provided a parameter selecting means for selecting a parameter relating to imaging and a recording means for recording the selected parameter in a storage medium in association with the captured image, the classification of the captured image is performed. Also, information effective for retrieval can be automatically recorded without bothering the operator.

According to a sixth aspect of the present invention, in the image pickup apparatus according to the fifth aspect, the information indicating the parameter relates to an image pickup scene including at least one of indoor / outdoor and weather. It is characterized by being.

According to the sixth aspect of the invention, it is possible to automatically record the imaged scene information that is effective in classifying and searching the imaged images, without the operator having to bother.

The invention according to claim 7 is the invention according to claim 5 or 6.
The image capturing apparatus described above is characterized in that the information indicating the parameter relates to an image capturing mode such as ON / OFF of the red-eye prevention function.

According to the seventh aspect of the invention, it is possible to automatically record the imaging mode information effective for classifying and searching the captured images, without the operator having to bother. In addition, when a person is the subject, there is a high possibility that the red-eye prevention function is turned on, so that the captured image can be determined as a person and processed.

The invention according to claim 8 is the image pickup device according to any one of claims 5 to 7, wherein the parameter for estimating whether or not the parameter selected by the parameter selecting means is correct. It is characterized by comprising a correctness estimation means and a correction means for correcting the information indicating the parameter when the parameter is estimated to be incorrect.

According to the invention described in claim 8, the information effective for classifying and searching the picked-up images can be automatically recorded without bothering the operator, and is automatically recorded. Since the obtained information can be determined and corrected, the accuracy of classification and retrieval of captured images can be improved.

According to a ninth aspect of the present invention, image processing is performed based on information stored in a storage medium in association with the image information by the image pickup apparatus according to any one of the first to eighth aspects. The method is characterized in that one or more of classification, retrieval, and display of image information is performed based on at least one piece of information stored in association with the image information.

According to the invention described in claim 9, the classification, retrieval or display of the picked-up images can be performed without using the image analysis by using the information recorded in association with the images. Therefore, primary classification and primary search of images can be performed efficiently in a short time.

According to a tenth aspect of the present invention, there is provided image information stored in a storage medium by the image pickup apparatus according to any one of the first to fourth aspects and information stored in association with the image information. An image processing method based on
When a person is included in the image information, the face of the person is automatically extracted from the image information based on the image information and the information of the moving body associated with the image information.

According to the tenth aspect of the present invention, since the line of the human face is automatically extracted using the image information and the information of the moving body associated with the image information, The amount of processing required for extraction is reduced, and the extraction can be performed at high speed.

The invention according to claim 11 is the image information stored in a storage medium by the image pickup device according to any one of claims 1 to 3 and stored in association with the image information, and An image processing method performed based on information about a moving body including information about a moving direction of the moving body, characterized by estimating a vertical direction of an image in image information from the information about the moving body.

According to the eleventh aspect of the invention, since the moving body information stored in association with the image information and including the moving direction information of the moving body is used, the image analysis can be performed again at the time of display. Even if not performed, the top and bottom of the image can be automatically and easily estimated.

The invention according to a twelfth aspect is an image processing method in a storage medium in which a database in which a plurality of image information is registered is recorded, and priorities are given based on information related to the image information in the database. An image processing method comprising: a priority adding step; and a storage step of storing the added information indicating the priority in association with the image data in a storage medium.

According to the twelfth aspect of the present invention, the priority level effective for the search can be automatically recorded without the operator's hand, and the search of the image data has a high priority level. Since the search can be performed sequentially, the search can be performed efficiently in a short time.

According to a thirteenth aspect of the present invention, in the image processing method according to the twelfth aspect, in the priority adding step, the size of the image, the photographer, and the mail are attached as the information related to the image information. The priority is given by using at least one piece of information among the number of times of printing, the number of times of displaying, and the number of times of printing.

According to the thirteenth aspect of the present invention, it is possible to automatically obtain the priority level effective for the search without bothering the operator.

The invention according to claim 14 is the image processing method according to claim 12 or 13, wherein in the priority adding step, the information related to the image information includes a plurality of items, and among the items, The above two or more quantifiable items are used as the criteria for determining the priority, and the quantized values of each item are weighted and added to give priority.

According to the fourteenth aspect of the present invention, a weight is added to some of the judgment criteria to give a priority, so that the image data can be searched by comprehensively considering a plurality of different judgment items. .

The invention of claim 15 is the same as claims 12 to 1.
4. The image processing method according to any one of 4 above, wherein in the priority addition step, a priority selection step capable of arbitrarily selecting at least one of the number of priority levels, the order of arrangement, and the weighting coefficient is included. It is characterized by having.

According to the fifteenth aspect of the present invention, the intention of the worker can be reflected in the image data search using the priority.

The invention according to claim 16 is the invention as claimed in claims 9 to 15.
The image processing method according to any one of, wherein at least one of information stored in association with the image information is used to classify, search, and display the captured images. It is characterized by doing one or more.

According to the sixteenth aspect of the present invention, the classification, retrieval or display of the picked-up images can be performed without using the image analysis by using the information recorded in association with the images. Therefore, primary classification and primary search of images can be performed efficiently in a short time.

According to a seventeenth aspect of the present invention, the image information submitted by the customer is automatically classified, and the information indicating the classification item used for the classification is stored in association with the image information. An image processing apparatus that handles image information provided by an image information classification service that provides information related to the image information to customers via a network or a storage medium, and is stored in association with the image information and the image information. Storage means for storing the stored information, display means for displaying the image information stored in the storage means, and correction means for correcting the information stored in the storage means.

According to the seventeenth aspect of the invention, for example,
When printing the film, analog image data, digital image data submitted by the customer, or when converting the image submitted by the customer into a digital image and returning it to the customer, the customer can easily add a classification item to the digital image. You can sort and search images. Further, since the correction means for correcting the information stored in the storage means is provided, the classification can be corrected according to the subjectivity of the operator,
Customers' needs can be reflected in the classification.

The invention according to claim 18 is an automatic classification step for automatically classifying image information submitted by a customer,
A storage step of storing information used for classification in the automatic classification step in association with the image information, and a step of storing the image information and the information associated with the image information and used for classification to a customer via a network or a storage medium. It is an image information classification service method characterized in that it is provided as a service.

According to the eighteenth aspect of the present invention, when printing the film, analog image data, or digital image data submitted by the customer, or when converting the image submitted by the customer into a digital image and returning it to the customer, When the image is digitized, the images are primary-classified almost at the same time, and the digitized image can be provided with the classification items used for the primary classification, so that the customer can easily perform the image classification and the search.

[0040]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the following embodiments do not limit the invention according to the claims, and
Not all the combinations of features described in the embodiments are necessary for the solution of the invention.

[First Embodiment] Hereinafter, a preferred embodiment to which the present invention is applied will be described in detail with reference to FIGS.

FIG. 1 is a block diagram showing the internal structure of the digital still camera 100 according to this embodiment.
As shown in FIG. 1, the digital still camera 100 includes a lens 1, a CCD 2, an A / D conversion unit 3, an image processing unit 4, an image memory 5, a control unit 6, a ROM 7, a RAM 8, an operation unit 9, and a display unit 10. , Card slot 11, strobe 12,
The CCD drive unit 13, the lens drive unit 14, and the like are provided.

The lens 1 forms an image of the imaging light of a subject.
The CCD 2 is connected to the CCD via the lens 1 within the pulse interval of the transfer pulse output from the CCD drive unit 13 described later.
The image pickup light of the subject formed on the image pickup surface (not shown) 2 is photoelectrically converted, that is, converted into a charge amount (hereinafter referred to as an analog signal) according to the received light amount. The CCD 2 outputs this analog signal to the A / D converter 3. The A / D converter 3 converts the analog signal input from the CCD 2 into a digital signal.

The image processing section 4 includes a process circuit and a compression circuit (not shown). The process circuit converts the digital signal input from the A / D converter 3 into a digital image signal such as a luminance signal and a color difference signal. Furthermore, the image processing unit 4
Performs various image correction processes such as edge adjustment, saturation adjustment, contrast adjustment, and noise removal processing on the digital image signal according to the instruction signal transmitted from the control unit 6. The compression circuit compresses the digital image signal after various image corrections based on a compression format such as JPEG (Joint Photographic Experts Group).

The image processing section 4 also includes a brightness level estimation circuit and a focus evaluation circuit (not shown). The brightness level estimation circuit estimates the brightness level based on the brightness signal output from the process circuit, and the focus evaluation circuit extracts a high frequency component from the digital image signal output from the process circuit.

The image memory 5 temporarily stores the digital image signal input from the image processing section 4. The image memory 5 can store a plurality of digital image signals here.

When the power is turned on by the operation of the power switch 9b, the control section 6 is supplied with power from a battery (not shown) and reads out and executes a main control program for controlling the operation of the digital still camera 100 from the ROM 7 described later. . Further, the control unit 6 controls the shooting mode setting button 9c.
The shooting mode is set in accordance with the shooting mode instruction signal input via, and the shooting process is performed based on the set shooting mode. The setting data of each shooting mode is stored in the ROM 7 as a setting data table (not shown).

The control unit 6 also stores various data generated when these programs are executed in a predetermined storage area (not shown) of the RAM 8.

The control unit 6 controls the CCD drive unit 13 to the CCD.
The shutter speed is adjusted by changing the number of charge sweep pulses output to 2. In addition, the control unit 6
Changes the output of the CCD 2 by the CCD drive unit 13 to C
Adjust the sensitivity of CD2.

The control section 6 has an AF (Auto Focus; automatic exposure adjustment) function and an AE (Auto Exposure: automatic exposure adjustment) function based on the imaging light taken in through the lens 1.
When the control unit 6 detects that the shutter button 9a is half pressed (hereinafter, this state is referred to as S1), the control unit 6 performs the AF and AE processing.

When the shutter button 9a is completely pressed (hereinafter, this state is referred to as S2) and the photographing instruction is input, the control section 6 sends the compressed digital image signal recorded in the image memory 5 to the card slot. It records on the memory card 11a stored in 11.

The control unit 6 executes the detection of the moving body shown in the flowchart of FIG. The detection of the moving body will be described in detail in the description of the operation of the digital still camera 100 described later. The control unit 6 further has a function of estimating whether or not the detected moving body is a person. The person estimating process and the moving body recognizing process will be described in detail in the description of the operation of the digital still camera 100 described later.

A ROM (Read Only Memory) 7 stores in advance various control programs executed by the control unit 6 and various data necessary for executing the programs. R
The OM 7 stores, in particular, programs for performing the moving body detection processing, the person estimation processing, and the moving body recognition processing shown in the flowchart of FIG. Further, the program can perform the same operation even if stored in the memory card 11a.

Further, the ROM 7 is used for moving object detection processing.
Stores reference data necessary for estimating image-related data, such as threshold value data for estimating a value D (x, y) for obtaining a moving pixel, which will be described later, data for estimating a skin color ratio in human estimation. To do.

A RAM (Random Access Memory) 8 has a program storage area and a data storage area (not shown). When executing the various processing programs, the control unit 6 expands the programs in the program storage area and temporarily stores the data generated when the various processing programs are executed in the data storage area.

The operation section 9 outputs a shutter button 9a for outputting a photographing instruction, a power switch 9b for outputting a power ON / OFF instruction, a photographing mode and an instruction for setting the selected photographing mode. A shooting mode setting button 9c, a shooting scene setting button 9d for selecting a shooting scene and outputting a selection result, a release button 9e for outputting a set shooting mode, a shooting scene release instruction, and the like are provided.

The display unit 10 is an LCD for displaying a photographed image.
A (Liquid Crystal Display) monitor 10a is provided. L
The CD monitor 10a has a function as a finder that continuously displays captured images captured by the CCD 2. In the shooting standby state, each pixel data of an image captured by the lens 1 and the CCD 2 every 1/30 seconds is subjected to predetermined signal processing by each circuit of the A / D conversion unit 3, and then the image memory 5 Is stored in the LCD monitor 1 and transferred to the display unit 10 via the control unit 6.
0a is displayed. Thereby, the operator can visually recognize the subject image from the image displayed on the LCD monitor 10a.

The card slot 11 is a memory card 11
a can be stored, and the stored memory card 11a and the control unit 6 are electrically connected. The memory card 11a is
It has a built-in flash memory (flash EEPROM) and records digital image signals taken by the digital still camera 100. Further, the strobe 12 illuminates the subject at the time of shooting.

The CCD drive section 13 controls and outputs the number of charge sweep pulses to the CCD 2 based on an instruction from the control section 6. In addition, the CCD driver 13
Increases or decreases the output of the CCD 2 based on an instruction from the control unit 6.

The lens drive unit 14 moves the lens 1 based on the instruction transmitted from the control unit 6 to adjust the in-focus position from the closest distance to the infinite distance.

Next, the operation of the digital still camera 100 will be described with reference to FIG. FIG. 2 is a flowchart illustrating a moving body detection process in the digital still camera 100.

First, the control section 6 estimates whether or not the photographing mode has been set by pressing the photographing mode setting button 9c (step S10).

When the photographing mode is set in step S10 (step S10; Yes), the control section 6 obtains a background motion vector as follows (step 11). That is, in the imaging standby mode, as shown in FIG.
As shown in, each pixel data of the image captured every 1/30 seconds is subjected to predetermined signal processing by each circuit of the A / D conversion unit 3, and then stored in the image memory 5. When the image capturing standby mode is entered after capturing the target image, the control unit 6 determines the target image and the image captured and stored in the image capturing standby mode immediately before capturing the target image (hereinafter referred to as the previous image). And the background motion vector is obtained. As a method of obtaining the background motion vector, the control unit 6
Is, for example, background motion vectors DX and D indicating how many pixels have been moved in the horizontal and vertical directions by performing pattern matching on the target image using the immediately previous image as a template
Y (unit is the number of pixels) is calculated. At this time, in order to reduce the processing time, it is preferable to use the images immediately before and the target image that have been reduced. At this time, the control unit 6
The matching degree is compared with a predetermined value stored in the ROM 7, and if it is lower than the predetermined value, it is determined that the digital still camera 100 has been largely moved, and the subsequent processing is not performed.

If the degree of matching is higher than the predetermined value, the control unit 6 determines, in step 12, that the immediately preceding image is DX, D.
After moving by Y pixels, a difference D (x, y) between the pixel and the corresponding pixel of the target image is obtained based on the following formula. D (x, y) = | R (x, y) -R '(x-Dx, y-
Dy) | + | G (x, y) -G '(x-Dx, y-D
y) | + | B (x, y) -B '(x-Dx, y-Dy)
| X and y respectively indicate horizontal and vertical coordinates (unit is the number of pixels). R (x, y), G (x, y) and B
(X, y) indicates the pixel values of R, G, and B at the (x, y) coordinates of the target image, respectively. Also, R '(x, y),
G '(x, y) and B' (x, y) indicate the pixel values of R, B, and G at the (x, y) coordinates of the immediately preceding image.

When the value of the pixel value difference D (x, y) between the target pixels is obtained in step 12, the control section 6
Proceeding to step 13, using the difference, an area that is relatively moving in the target image is obtained. Step 13
A description will be given with reference to FIG. 4 showing a flow for obtaining a relatively moving area.

Compared with a predetermined threshold value stored in the ROM 7, if the value of D (x, y) exceeds the threshold value, 1,
If it does not exceed 0, create a binary image with 0
An area exceeding a predetermined threshold is obtained by performing labeling on the value image. As a result, a region corresponding to the “differential labeling” in FIGS. 5, 7 and 8 is extracted, and this extracted region is set as a motion candidate region. There are three cases in which such a motion candidate area is detected. When the moving area in the previous image and the moving area in the target image overlap, when a moving area not in the previous image appears in the target image, and This is the case where the moving area in the image is separated from the moving area in the target image. A processing method in such a case will be described based on the flow shown in FIG.

In step 131, the control section 6 uses the difference between the target images obtained in step 12 to perform labeling as a motion candidate area.

After labeling, the control section 6 proceeds to step 13
In step 2, pattern matching is performed on the target image using the region of the immediately preceding image corresponding to the labeled label region of the target image as a template, and the highest matching degree value is calculated.

At step 133, the control section 6 estimates whether the matching degree is larger than a predetermined threshold value stored in the ROM 7.

When the matching degree is larger than the threshold value,
The control unit 6 proceeds to step 134 and estimates whether the motion candidate area in the immediately preceding image and the motion candidate area in the target image overlap or are separated. If both motion candidate areas overlap, the control unit 6 determines in step 13
5, the difference between corresponding pixels in the immediately preceding image and the target image is calculated at the position with the highest degree of matching, and the ROM 7
It is determined that a region lower than the predetermined threshold value stored in is overlapping, and this region is newly estimated as a motion region. When both motion candidate areas do not overlap, the control unit 6 proceeds to step 136 and estimates that the area of the target image corresponding to the template area of the immediately previous image is not the motion area. Here, the moving direction of the template area of the immediately preceding image is the moving direction of the moving area.

When the degree of matching is smaller than the predetermined threshold value, the motion candidate area appears or disappears in the target image (moving area goes out of the frame).
Either of them. In this case, the control unit 6 estimates the motion candidate area as a motion area. When the motion candidate region disappears, it is estimated as a motion region at this point, but is not estimated as a motion region in the next estimation.

In step 137, the above steps 132 to 136 are performed for all the motion candidate areas obtained in step 131, and the coordinates, number and moving direction of the motion areas are obtained. Note that, for example, when the number of motion areas is n, the coordinates of the motion area are the coordinates of the center of the area (Xi, Yi) and the width of the area (WIDTHi).
And the height of the region (HEIGHTi) (i is 1 to n). The moving direction may be indicated by (DXi, DYi).

Here, an example in which a moving area is obtained from adjacent images in still images taken at 1/30 second intervals has been shown, but for every nth image, regardless of the adjacent images. You may make it process.

The control unit 6 records the coordinates, the number, and the moving direction of the moving areas thus obtained in association with the reference image in the memory card 11a which is a storage medium. As a recording method to record in association with the image,
It is preferable to write at a predetermined position in the header, such as Exif. Further, another method of recording the information of the moving area as a separate file in association with the image file may be used. In addition, although a memory card is used as the storage medium in the description, for example, a compact flash (registered trademark) that can be attached to and detached from the image pickup device,
Other storage media such as an SD card may be used.

The control section 6 performs the following image processing by using the image information recorded in the storage medium as described above. The control unit 6 opens the recorded image, first reads the number of moving areas, and then reads the center coordinates, width, and height of the areas by the number, and obtains each area. The control unit 6 determines whether the top and bottom of the image are vertical or horizontal based on the moving directions of these areas. The control unit 6 may have a function of estimating whether or not a person is included in the image. When the moving area is estimated to be a person, the top and bottom of the image are up and down in the area surrounding the person. In some cases, if the top and bottom of the area and the top and bottom of the image are left and right, face estimation is performed for the left and right parts of the area to estimate whether or not a face is included in those areas (FIG. 10). reference). It should be noted that the estimation of whether or not a person is included can be estimated by a general method such as whether or not the skin color is included, and the face estimation is such that the skin color is included in a predetermined proportion or more, the eyes are included in the skin color area, It can be done by a general method such as whether the mouth is included. In this way, the control unit 6 can extract the face part. Further, since the face estimation is performed in an extremely limited area, the processing amount of the face estimation can be reduced,
Face extraction can be performed at high speed. By using the position information of the face area thus extracted, the top and bottom of the image can be accurately determined. Therefore, it is preferable to record the top and bottom information of the image determined here in association with the image. .

As described above, by performing image analysis using moving image information, it is possible to cancel the movement of the background caused by moving the camera, and it is possible to accurately obtain the area in which the moving body is moving. . Further, since such a moving area is highly likely to be a person, it is possible to easily and quickly extract the area in which the person is shown, the number of persons, and the moving direction. Further, since it is sufficient to estimate whether or not the moving area is a person and extract the face of the person not only for the entire image but only for the moving area, the processing required for the estimation can be performed at high speed. Can be done at.

<Modification 1> Hereinafter, Modification 1 of the first embodiment will be described. Modification 1 Digital Still Camera 1
At 00, the control unit 6 obtains D (x, y) similarly to the first embodiment, labels the image, obtains the motion candidate region, and then determines that the motion candidate region in the target image is human. It has a function of estimating whether there is any and a function of recognizing a motion candidate region estimated to be a person as a motion region.
The operation of the digital still camera 100 will be described. The control unit 6 calculates the difference and labels the image similarly to the first embodiment, and after calculating the motion candidate region,
Estimating whether a person is included in the motion candidate region,
The estimation result is associated with the image and the memory card 11
Record in a. It is possible to estimate whether or not the motion candidate area is a person by using a general method such as estimating whether or not the target area contains a predetermined amount of skin color. In this way, by limiting the motion area to people, it is possible to reduce processing such as classification and search.

<Modification 2> In the digital still camera 100 of Modification 2, the control unit 6 obtains D (x, y) as in the first embodiment, labels the image, and determines the motion candidate region. After obtaining, the motion candidate area is simply set as a motion area, and the position and the number thereof are recorded in association with the image. Such a digital still camera 10
The operation of 0 will be described. Similar to the first embodiment, the control unit 6 obtains a difference, labels an image, obtains a motion candidate region, estimates the motion candidate region as a motion region, and estimates the estimation result as The image is recorded in the memory card 11a in association with the image. In this way, since the process for estimating the motion candidate region as the motion region can be omitted, the control unit 6 can perform the process using the information of the moving body at high speed.

[Second Embodiment] First, the structure will be described. In the second embodiment, the digital still camera 100 replaces the moving object detection means with the temperature distribution in the imaged object in the imaged image. It is equipped with a desired temperature detecting means. More specifically, the digital still camera 100 is
The control unit 6 has a sensor for detecting infrared rays, and has a function of receiving and processing information transmitted as an electric signal from the infrared sensor.

Next, the operation of the digital still camera 100 will be described. The sensor detects infrared rays radiated from an object and sends it to the control section 6 as an electric signal. The control unit 6 obtains a temperature distribution in the image from the received infrared information, and an area in which a person is photographed from the obtained temperature distribution,
And the number of people. For example, the control unit 6 binarizes and labels the image of the temperature distribution depending on whether the temperature is in the range of 30 ° C. to 40 ° C., obtains a candidate region in which a person is captured, and the person is captured in this candidate region. To estimate whether
The estimation result is recorded in the memory card 11a in association with the image. As described above, since the temperature information is used instead of the moving image information, the control unit 6 limits the area in which the process for extracting the person is performed without performing the pattern matching to obtain the area in which the person is captured. It is possible to extract a human area at high speed.

The control unit 6 may be provided with an estimating means for estimating whether or not the candidate area of a person is a person based on the ratio of the skin color and the like as described in the first embodiment.
With such an estimation means, it is possible to more accurately extract a region in which a person is shown. Further, as in the first embodiment, a moving body detecting unit may be provided together with the temperature detecting unit, and the image may be analyzed by combining the temperature information and the moving image information. By doing so, the area of the person can be more accurately extracted, and the top and bottom of the captured image can be estimated. Although an infrared sensor is used here as a method for detecting the temperature, the method is not limited to this.

[Third Embodiment] In the digital still camera 100 according to the third embodiment, when an image is taken,
The operator presses the shooting scene button 9d of the operation unit 9 to select a shooting scene from the taken scenes such as indoors, outdoors, people, landscapes, night views, and weather, and also, the shooting mode setting of the operation unit 9 is performed. The button 9c is pressed to select ON / OFF of the red-eye prevention function, and support for setting the selected photographing mode is output to the control unit 6. It should be noted that a plurality of shooting scenes and shooting modes may be selected, and the shooting scenes and shooting scenes are not limited to these. The shooting scene and shooting mode thus selected can be erased and reset by pressing the release button 9e of the operation unit 9. The selected shooting scene and shooting mode are associated with the captured image in the memory card 11 in the control unit 6.
recorded in a.

Further, it is preferable that the control unit 6 has a means for correcting the imaged scene input by the operator based on the information and the image pickup mode obtained by the image analysis by the control unit 6. Specifically, whether the shooting scene is a person or a landscape may be determined from the number of persons in the image if the number of persons in the image is recorded in association with the image. Further, it is determined whether the number of persons is within a predetermined number stored in the ROM 7, and if the number of persons in the image is within the predetermined number, it is estimated that the image is a portrait image. If it is not within the predetermined value, it may be estimated that it is a group photo. In addition, it may be assumed that the person is a person when the photograph is taken with the red-eye prevention function, which is a photographing mode, set to ON. Based on this estimation result, the control unit 6
It is recorded in the memory card 11a in association with the captured image. Further, the control unit 6 compares the average brightness of the entire image (specifically, for example, the average value of R + G + B) with a predetermined threshold value stored in the ROM 7, and determines that the average brightness of the image is
If it is lower than the threshold value, it may be determined as a night view, and the determination result may be recorded in the memory card 11a. In addition, the control unit 6 performs image analysis, estimates whether the light source is a fluorescent lamp or tungsten, for example, based on color temperature, and determines that it is indoors when the light source is a fluorescent lamp or tungsten. If the light source is neither a fluorescent lamp nor tungsten, it may be judged to be outdoors and recorded in the memory card 11a. Also, by connecting the digital still camera 100 to the Internet, or
After shooting, read the image data into a computer connected to the Internet, search the weather on the Internet from the shooting position and date recorded in association with the image,
You may make it collate with the weather set at the time of photography.

By reading the photographic scene information and the photographic mode information thus recorded together when the image is read, the images can be classified without performing image analysis. Also, when performing a search, if a primary search is performed according to the shooting mode information and a more advanced search is performed by image analysis after limiting the search range, the processing required for the search can be reduced and the search can be performed at high speed. I can.

[Fourth Embodiment] A digital still camera 201 as shown in FIG. 11 has an input section 202 including an operation button for setting a photographing mode, turning on / off a power source, operating a shutter, and the like. The image pickup unit 205 and the image pickup unit 20 which pick up an image according to the setting of 202
CP for processing images and image information taken by
A U204, an application 203 for reading information required by the CPU 204 in image processing, and a non-volatile RAM 206 for recording a processed image and image information.
As shown in FIG. 12, an image processing apparatus 301 that reads and processes an image and image information recorded in the nonvolatile RAM 206 has an input unit including operation buttons for selecting an image processing mode, selecting a priority, and the like. 303, input unit 30
A CPU 302 that processes an image and image information read according to the setting of No. 3, an album application 305 that reads information required by the CPU 302 in image processing,
A calculation memory 304 used for image processing in the CPU 302 is provided. The image processing apparatus 301 also includes a display unit (not shown) that displays the operation status and selection items.
In the image processing apparatus 301, the CPU 302 has a function of automatically giving priority to image data in the database and recording the priority in the header of the image file or the recording medium. For example, the digital still camera 201
Can pick up an image by selecting an image size mode such as Super Fine, Fine, Economy.
In No. 2, the priority corresponding to the size of the image is associated with the image and is recorded again in the database. That is, for a large image captured by selecting Super Fine, a high priority is recorded in association with the image. Further, as described in the first embodiment, the number of persons in the image can be recorded in association with the image. However, in the image processing apparatus 301, when reading the image, the number of persons is read together, A high priority is associated with an image of a person and recorded. The number of people information may be automatically extracted in the image processing apparatus. In addition, the image is analyzed, the face of the person in the image is compared with the face of the person registered in advance, and the priority is given according to the face of which person. In addition, a digital still camera 201 capable of mail transfer and the digital still camera 2
When 01 is built in the mobile phone, the number of times an image is attached to an e-mail can be automatically recorded. However, an image having a large number of e-mail attachments is recorded with a high priority. When the digital still camera 201 is connected to a network or directly connected to a printer for printing, the number of times an image is printed can be automatically recorded. Images with a high order count are recorded with a high priority. For example, the following weighted addition is performed on the priority recorded in this way, and the priority incorporating a plurality of determination items is recorded in association with the image. Image size priority x 0.3 + number of prints x 0.2 + number of photographers x 0.
4 + priority according to the number of times attached to mail × 0.1 = total priority

As described above, the effective priorities for the retrieval can be automatically recorded without the operator's trouble, and the retrieval of the image data can be performed in the descending order of the priorities. The search can be efficiently performed in a short time. Furthermore, since weighting is added to some of the judgment criteria to give priority, it is possible to retrieve the image data by comprehensively considering a plurality of different judgment items.

The number of priority levels, the order in which they are arranged, and the weighting coefficient at the time of weighted addition are, for example, as shown in FIGS.
As shown in FIGS. 4 and 15, it may be possible for the operator to arbitrarily determine. That is, the CPU 302 has a function of preferentially processing an image to which a specific priority is added when searching for an image to which the priority is added as described above. As shown in FIG. 13, when the operation button of the input unit 303 is pressed, the image processing mode is entered, and from the plurality of image processing modes displayed on the display unit, when the operation button of the input unit 303 is pressed again, “automatic priority” is set. Degree ”to enter the priority judgment item mode. In the priority judgment item mode, a judgment item is selected and the priority step number mode is entered. The priority level is the priority divided by the number of levels.
For example, when the priority determination item is a photographed person, the faces of the worker himself / herself and the family are registered in advance, and the face extracted from the captured image is compared with the registered face image.
Based on the estimation result, the image showing the worker himself is given priority 1, the image showing the worker's family is given priority 2, the image showing the other person is given priority 3, and the person is shown. The number of priority levels is set such that the within-image is priority 4.

If the judgment item can be quantified by the operator's selection, the number of steps is determined. For example, the priority judgment item is the image size, and when the number of stages selected by the operator is 5, the average is divided into 5 sections according to the size from the maximum size to the minimum size of the image, and the images within each range are each divided by 1 Give priorities 5 to 5. If the priority determination item is a photographed person, it cannot be quantified, and thus the selection of the number of priority levels may be omitted, or the priority may be determined according to the imaged object as described above.

In the priority step number mode, the input unit 30
When the priority step number is selected by pressing the operation button No. 3, the order mode is entered, and “ascending order” or “descending order” is selected. The “ascending order” and the “descending order” are the order in which the priorities are arranged. When the priority judgment items can be quantified, the judgment is based on the amount. For example, when the operator selects “ascending order”, the priority of an image having a large size is set to be high. On the contrary, when “descending order” is selected, the priority of the image having a small size is set to be high. When the priority judgment item is the person to be photographed, it cannot be quantified, so that oneself->family-> others, or family->me-> others, etc. may be selected, and the above-mentioned priority is used. You may attach "ascending order" and "descending order".

When the order of priority is selected, the intention confirmation mode is entered, and the selected priority judgment item, the number of priority levels, and the priority are displayed. If there is no error in each selected item, the CPU 302 supports the image search with the selected setting by pressing the operation button of the input unit 303.
And the image is searched. Also, at this time,
When changing the selected item, the setting can be canceled or a new setting can be performed by pressing the operation button of the input unit 303. In addition, when there are a plurality of priority judgment items and the weighting addition is performed on the plurality of judgment items, the operator can specify the weighting coefficient of each judgment item, and the weighting coefficient is recorded in association with the image. The priority can be changed by the operator. By doing so, the intention of the worker can be reflected in the image data search using the priority. The image processing apparatus 301 capable of classifying and retrieving images in this way can display the retrieved image together with the image information on a display unit (not shown) or a connected display device such as a monitor or a television. As shown in FIG. 15, the image information such as the priority can be arbitrarily modified by pressing the operation button of the input unit 303 or the like. Therefore, classification,
In the search, the intention of the worker can be reflected more easily.

As described above, by searching the images in order of priority, the images with high priority can be displayed on the display unit so that the images can be easily searched for, and more advanced search can be performed by image analysis or the like. Even when performing the search, the search processing can be performed at high speed by searching only the images in the limited range of high priority.

Also, when deleting unnecessary images, the images of lower priority are automatically searched by the priority judgment item selected by the operator, and they can be deleted in a batch. It is more efficient because there is no need to specify one.

When displaying the priority on the display unit, each priority may be displayed in a specific color or code. By doing so, the priority can be understood at a glance as compared with the case where only the characters are displayed, so that the selection operation can be made easier.

Further, the above-mentioned automatic classification may be used to provide a negative image information classification service in a laboratory. The image processing apparatus 301 reads the image submitted by the customer, analyzes the image information recorded in association with the image such as the shooting mode and the shooting scene, and analyzes the image in the laboratory, and obtains it from the image analysis. Images are primarily classified by information. The image submitted by the customer need not be digital data, but may be analog data such as film. When the image is analog data, the image is digitized when the image is read. Further, the image processing device 301 classifies the images in a primary manner, associates the classification items used for the classification with the images, and records the images in a storage medium such as a CD-R or a DVD-R together with the images. Note that the image and the classification item associated with the image may be uploaded to a predetermined site so that the customer can browse them from the Internet, instead of recording them in a storage medium. Further, software for reading the classification items and automatically classifying and displaying the images is also recorded on the storage medium together with the images and provided to the customer, so that the customer can use a predetermined PC at home.
Alternatively, the classified images may be browsed on a monitor or a television screen using hardware such as a game machine. Further, it is preferable that the classification items recorded in association with the image can be arbitrarily corrected by the input unit 303 so that the subjectivity of the customer can be reflected. According to such a service, it is possible to realize automatic classification of images at high speed by using the information recorded in association with the images, and when a large number of negatives are digitized, they are already classified as primary. Since it is there, it is possible to reduce the trouble of organizing.

[0095]

According to the invention described in claims 1 to 11, 17 and 18, by recording the information effective for the search and classification in relation to the image, the search and the classification can be performed with high accuracy. Moreover, according to the invention of claims 12 to 16,
By giving priority to image data, some images can be processed preferentially, data processing such as search / browsing can be performed more efficiently, and the usability for workers is improved.

[Brief description of drawings]

FIG. 1 is a digital still camera 100 to which the present invention is applied.
3 is a block diagram showing the internal configuration of FIG.

FIG. 2 is a digital still camera 100 to which the present invention is applied.
7 is a flowchart illustrating a moving body region detection process in FIG.

FIG. 3 is a digital still camera 100 to which the present invention is applied.
3 is a schematic diagram showing an image captured in a shooting standby mode in FIG.

FIG. 4 is a digital still camera 100 to which the present invention is applied.
6 is a flowchart illustrating a process for obtaining a relatively moving area in FIG.

FIG. 5 is a schematic diagram showing extraction of a moving body region in the present invention.

FIG. 6 is a schematic diagram showing extraction of a moving body region in the present invention.

FIG. 7 is a schematic diagram showing extraction of a moving body region in the present invention.

FIG. 8 is a schematic diagram showing extraction of a moving body region in the present invention.

FIG. 9 is a schematic diagram showing extraction of a moving body region in the present invention.

FIG. 10 is a schematic diagram showing extraction of a region in which a human face is shown in the present invention.

FIG. 11 is a schematic diagram showing a configuration of a digital still camera according to the present invention.

FIG. 12 is a schematic diagram showing a configuration of an image processing apparatus according to the present invention.

FIG. 13 is a schematic diagram showing an operation procedure (worker assistance) in the digital still camera of the present invention.

FIG. 14 is a schematic diagram showing an operation procedure (manual correction) in the digital still camera of the present invention.

FIG. 15 is a schematic diagram showing an operation procedure (manual correction) in the digital still camera of the present invention.

[Explanation of symbols]

100 digital still camera 1 lens 2 CCD 3 A / D converter 4 Image processing section 5 image memory 6 control unit 7 ROM 8 RAM 9 Operation part 9a Shutter button 9b power switch 9c Shooting mode setting button 9d Shooting scene setting button 9e Release button 10 Display 10a LCD monitor 11 card slots 11a memory card 12 strobes 13 CCD driver 14 Lens drive 201 digital still camera 202 Input section 203 application 204 CPU 205 imaging unit 206 Non-volatile RAM 301 image processing device 302 CPU 303 Input section 304 arithmetic memory 305 Album application

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ identification code FI theme code (reference) H04N 5/225 H04N 5/225 C F 5/907 5/907 B 7/18 7/18 K (72) Inventor Takasei 2970 Ishikawa-cho, Hachioji City, Tokyo Konica Stock Company F-term (Reference) 5B075 ND06 NR02 NR12 PQ02 PR08 5C022 AA05 AA13 AB15 AC00 AC03 AC13 AC42 AC69 5C052 AA17 AB09 AC08 CC11 DD02 GA02 GA03 GA09 GB01 A09 501 CA04 CA05 DA06 EA07 FA02 FC01 FC13 GA04 GB06 HA18 5L096 BA02 CA02 EA39 FA67 HA04 HA08 JA09 KA09 LA04 LA05 LA11

Claims (18)

[Claims] 1. An image pickup means for performing image pickup and outputting image information obtained by the image pickup; and a moving body that is moving in a range picked up at the time of image pickup from the image information, and the moving body is A moving body detection unit that obtains and outputs information about one or more moving bodies out of information about an occupied area, information about the moving direction of the moving body, and information about the number of moving bodies; Storage means for associating the generated image information with information about the moving body obtained from the image information by the moving body detection means, and storing the image information and the moving body information in a storage medium. An image pickup device characterized by. 2. The moving body detecting means uses a plurality of pieces of image information which are picked up by the image pickup means at different times and whose image pickup ranges overlap each other, so as to detect an image portion which moves relatively to an image portion which becomes a background. The image pickup device according to claim 1, wherein the image pickup device detects the moving object. 3. An estimation means for estimating whether or not the moving body detected by the moving body detecting means is a person, and when the estimating means estimates the moving body as a person, the memory is stored. 3. The image pickup device according to claim 1, wherein the means adds information that the moving body is a person to the information of the moving body stored in the storage medium. 4. An image pickup means for picking up an image and outputting the image information obtained by the image pick-up, a temperature detecting means for outputting the temperature information obtained by detecting the temperature within the picked-up range, and the temperature information. Alternatively, it is estimated whether or not a person is included in the image information from the temperature information and the image information, and when it is estimated that a person is included in the image information, the area occupied by the person in the image information is Information, information of the moving direction of the person, person extraction means for obtaining and outputting information of any one or more of the number of people information, image information imaged by the imaging means, An image pickup apparatus comprising: a storage unit that associates the person information obtained from the image information by the person extraction unit and stores the image information and the person information in a storage medium. 5. An image pickup means for picking up an image and outputting image information obtained by the image pickup, a parameter selecting means for selecting a parameter relating to the image pickup, and information indicating the parameter selected by the parameter selecting means in a picked-up image. An image pickup device comprising: a recording unit that stores the data in a storage medium in association with each other. 6. The image pickup device according to claim 5, wherein the information indicating the parameter relates to an image pickup scene including at least one of indoor and outdoor and weather. 7. The image pickup device according to claim 5, wherein the information indicating the parameter relates to an image pickup mode such as ON / OFF of a red-eye prevention function. 8. A parameter correct / incorrect estimation unit for estimating whether or not the parameter selected by the parameter selection unit is correct, and a correction unit for correcting information indicating the parameter when the parameter is estimated to be incorrect. The image pickup device according to claim 5, further comprising: 9. An image processing method performed based on information stored in a storage medium in association with the image information by the image capturing apparatus according to claim 1. An image processing method characterized by performing at least one of classification, retrieval, and display of image information based on at least one of the information stored in association with. 10. An image performed by the image pickup device according to claim 1 based on image information stored in a storage medium and information stored in association with the image information. A method of processing, wherein based on the image information and information on the moving body associated with the image information, when a person is included in the image information, automatically extracting a face of the person from the image information. An image processing method characterized by: 11. The image information stored in a storage medium by the image pickup device according to claim 1 and the image information stored in association with the image information, and the moving direction of the moving body. An image processing method performed on the basis of information on a moving body including information on the moving body, wherein the vertical direction of the image in the image information is estimated from the information on the moving body. 12. An image processing method in a storage medium in which a database in which a plurality of image information is registered is recorded, and a priority addition step of assigning a priority based on information related to the image information in the database, A storage step of storing the added information indicating the priority in the storage medium in association with the image data. 13. In the priority adding step, as information related to the image information, at least one of image size, photographed person, number of times attached to mail, number of times displayed, number of times printed 13. The image processing method according to claim 12, wherein priorities are given using the above information. 14. In the priority adding step, the information related to the image information includes a plurality of items, and two or more digitizable items of the items are used as a criterion for determining the priority. 14. The image processing method according to claim 12, wherein the numerical values of each item are weighted and added to give priority. 15. The priority adding step comprises a priority selecting step capable of arbitrarily selecting at least one of the number of priority levels, the order of arrangement, and the weighting coefficient. The image processing method described in any one of 1. 16. One or more of classification, retrieval, and display of the captured image is performed by using at least one or more of information stored in association with the image information. The image processing method according to claim 9. 17. Image information submitted by a customer is automatically classified, information indicating classification items used for the classification is stored in association with the image information, and the information is associated with the image information and the image information. An image processing apparatus that handles image information provided by an image information classification service that provides the customer with the information via a network or a storage medium, and stores the image information and the information stored in association with the image information. An image processing apparatus comprising: a unit, a display unit that displays the image information stored in the storage unit, and a correction unit that corrects the information stored in the storage unit. 18. An automatic classification step of automatically classifying image information submitted by a customer, a storage step of storing information used for classification in the automatic classification step in association with the image information, and the image information. And an image information classification service method, characterized in that the information related to the image information and used for classification is provided to a customer via a network or a storage medium.