JP5024093B2 - Image capturing device, display device thereof, and image data management system - Google Patents

Description

  The present invention relates to an image capturing device, a display device thereof, and an image data management system.

In recent years, with the widespread use of digital cameras, it has become common to take many photographs. In such situations, it is important that characteristic images can be easily extracted from the photographed images (photos). It is. For this reason, meta information at the time of shooting an image, for example, shooting date and time, shooting location, shooting conditions (for example, shutter speed, aperture value), and the like are added to the shot image. This meta information is generally integrated into image data in the data format of the Exif standard. Therefore, if the meta information added to the image is used, the target image can be searched quickly or efficiently. It can be carried out.

  Further, as part of the above-described meta information, it is also considered to add biometric information (or psychological information) of the photographer when the image is taken, such as a heart rate and a respiration rate. When this biometric information is included, it is possible to refer to the biometric information of the photographer at the time of photographing, so that an image photographed in a special state can be searched more accurately.

  In addition, research on a life log that records all of your actions by always wearing a camera and automatically shooting at predetermined time intervals is also being conducted. If information or psychological information is used, this is facilitated when an important image is searched from an enormous number of images.

For example, there are the following prior art documents.
JP 2005-124909 A JP-A-2005-141281 Japanese Patent Laid-Open No. 2005-250977

  The present invention is intended to add biometric information or action information as meta information to an image and to make effective use of the meta information when searching for an image or the like.

In this invention,
In a shooting device that takes a picture of a subject and obtains image data,
A biometric information sensor that detects biometric information of the photographer at the time of photographing the subject;
A behavior information sensor for detecting the photographer's behavior information at the time of photographing,
At the time of shooting, the biological information detected by the biological information sensor and the behavior information detected by the behavior information sensor are stored in association with the image data as meta information of the photographer at the time of shooting. An image capturing apparatus is provided.

  According to this invention, since the photographer's biometric information and behavior information are added to the photographed image as more sensitive data, the image photographed in a special state is different from other images. It can be displayed separately.

[1] Example of Digital Still Camera [1-1] Example of Configuration FIG. 1 shows an example of a digital still camera to which the present invention can be applied. That is, reference numeral 10 denotes the entire digital still camera, and an image of the subject SBJT is projected onto an image sensor, for example, a CCD image sensor 12 by the imaging lens 11, and a video signal of the subject SBJT is extracted from the CCD image sensor 12. Then, this video signal is supplied to the analog processing circuit 13 to be a video signal subjected to preprocessing such as CDS processing and AGC processing, and this video signal is supplied to the A / D converter circuit 14 to be converted into a digital video signal. A / D conversion is performed.

  The digital video signal is supplied to the digital processing circuit 15 and various digital corrections for improving the image quality are performed. Then, the digital video signal is supplied to the display control circuit 16 and D / A converted into an analog video signal. To the viewfinder 17. Thus, at this time, the subject SBJT photographed by the imaging lens 11 is displayed on the finder 17 in a moving image state, and the photographing content is monitored.

  Reference numeral 20 denotes a microcomputer for system control, which includes a CPU 21 for executing various programs, a ROM 22 for storing various programs, a RAM 23 for a work area, and information unique to this camera. And a non-volatile memory 24 for storing information set by the user.

  Further, the microcomputer 20 has a communication circuit 25, and the communication circuit 25 can upload / download data to / from an external server (not shown) through the connector 26. Various operation buttons 27 such as a shutter button are connected to the microcomputer 20.

  A control signal for each processing is supplied from the microcomputer 20 to the processing circuits 13, 15 and 18, and a predetermined control signal is also supplied to the timing generator 41 to form various timing signals. A timing signal is supplied to the CCD image pickup device 12 to realize image pickup in the CCD image pickup device 12.

  In addition, a biometric information sensor 31 that detects, for example, a heart rate and an electrical resistance (impedance) of the skin is prepared for detecting the biometric information of the photographer, and the detection output thereof is taken into the microcomputer 20 through the connector 35. In this case, the biological information sensor 31 is configured, for example, as a wristwatch and is attached to the wrist of the photographer, and detects the heart rate of the photographer and the electrical resistance of the skin. The amount of sweating can be known from the electrical resistance of the skin detected by the biological information sensor 31. Therefore, the biological information of the photographer can be obtained by analyzing the detection output of the biological information sensor 31.

  Further, the digital still camera 10 is provided with an acceleration sensor 32, and an acceleration applied to the digital still camera 10 when the photographer is walking with the digital still camera 10 in his hand or from the neck. Is detected, and the detection output is also taken into the microcomputer 20. By analyzing the acceleration detection output, the photographer is in a state of walking slowly, walking fast, stopping, getting on a vehicle such as a bus or train, for example, The photographer's action information can be detected.

  The microcomputer 20 is connected with a clock circuit 33 and a position information sensor such as a GPS unit 34. In this case, the clock circuit 33 is for obtaining information of the photographing date and time when the subject SBJT is photographed, and the GPS unit 34 is for obtaining position information of the photographing location.

  Further, a predetermined control signal is supplied from the microcomputer 20 to the timing generator 41 to form various timing signals, and this timing signal is supplied to the CCD image pickup device 12 to realize photographing in the CCD image pickup device 12. Further, various control signals and data are exchanged between the analog processing circuit 13 and the digital processing circuit 15 and the microcomputer 20.

[1-2] Example of Operation When the power of the digital still camera 10 is on, the image of the subject SBJT photographed by the CCD image sensor 12 is displayed on the finder 17 as a moving image as described above. . When the shutter button of the operation buttons 27 is pressed, the digital video signal (still image video signal) at this time is taken into the microcomputer 20 from the digital processing circuit 15 and is compressed by, for example, the JPEG method. This image data is temporarily stored in the RAM 23. That is, a still image is shot, and the image data is temporarily stored in the RAM 23 as shown in FIG. 2A, for example.

  Further, at the time of shooting, data of shooting conditions such as a shutter speed and an aperture value set by the microcomputer 20 are supplied to the RAM 23 as shooting information. At this time, the shooting date / time data indicated by the clock circuit 43 is taken into the microcomputer 20 and is used as a part of the shooting information.

  In this way, as shown in FIG. 2A, in the RAM 23, the data of the photographing condition at this time is added to the image data stored in the RAM 23, for example, in the Exif format.

  Furthermore, the heart rate and electrical resistance data detected by the biological information sensor 31 and the acceleration data detected by the acceleration sensor 32 are supplied to the RAM 23, and the position information data of the shooting location indicated by the GPS unit 44 is provided. The image data supplied to the RAM 23 and stored therein is added in, for example, the Exif format. Note that the data detected by the biometric information sensors 31 and 32 can be data for a predetermined period, for example, ± 30 seconds, with respect to the time (instant) when the shutter button is pressed.

  Thus, as shown in FIG. 2A, the image data stored in the RAM 23 in the RAM 23 includes the shooting information, the heart rate and electrical resistance of the photographer, the acceleration, and the shooting position data in the Exif format, for example. Added.

  Then, the image data to which each data is added is supplied to the recording / reproducing circuit 18 through the digital processing circuit 15 and recorded on a non-volatile recording medium such as a memory card 19. When an arbitrary subject is photographed in this way, information such as the photographer's biological information and behavior information, photographing date and time, and photographing position is added to the image data and recorded in the memory card 19.

[2] Example of Display Device [2-1] Example of Configuration In this example, a personal computer is used as the display device and a predetermined program corresponding to [1] is executed.

  In FIG. 3, reference numeral 60 denotes a personal computer. The personal computer 60 includes a CPU 61 for executing various programs, a non-volatile memory 62 in which an initial program loader is stored, a work area RAM 63, and a large capacity. And a hard disk drive 64 as a storage device. The nonvolatile memory 62, the RAM 63, and the hard disk drive 64 are connected to the CPU 61 through the system bus 69.

  In this case, the hard disk drive 64 is provided with an OS and various application programs, and an analysis program and a display program corresponding to [1] are prepared as part of the application programs. Yes.

  Further, an input device 65, a card reader 66, and a communication circuit 67 are connected to the system bus 69. In this case, the input device 65 is for a user to input instructions and data, such as a keyboard and a mouse, and the card reader 66 is inserted with a memory card, for example, the memory card 19 of FIG. When this is done, the data is accessed with the memory card 19. The communication circuit 67 is for uploading / downloading data to / from an external network or server through the connector 68.

  Further, the personal computer 60 is provided with a display control circuit 71 and an audio reproduction circuit 73, which are also connected to the system bus 69. The display control circuit 71 has a video RAM (not shown), and display data is supplied to the video RAM, and it is repeatedly read out at a predetermined cycle and converted into a video signal. 72 to display an image.

  Further, when the digital audio data is supplied to the audio reproduction circuit 73, the audio reproduction circuit 73 performs a decoding process such as MP3 as necessary, further D / A converts the digital audio data into an analog audio signal, and outputs the analog audio signal. An audio signal is supplied to the speaker 74.

[2-2] Example of Operation [2-2-1] Analysis of Biological Information and Behavior Information When the personal computer 60 is turned on, the CPU 61 executes the initial program loader of the nonvolatile memory 62 and starts from the hard disk drive 64. The OS is read out, transferred to the RAM 63, executed, and waits for execution of the application program.

  Therefore, the memory card 19 storing the image data (FIG. 2A) is inserted into the card reader 66 by [1], and the image data stored in the memory card 19 is transferred to the hard disk drive 64.

  Then, by operating the input device 65 and executing the analysis program prepared in the hard disk drive 64, the photographer's heart rate and electrical resistance added to the image data (FIG. 2A) are analyzed. The photographer's biological information is extracted, and the acceleration is analyzed to extract the photographer's action information.

  As described above, it is possible to know the amount of sweat from the electrical resistance of the skin, and thus it is possible to obtain the photographer's biological information by analyzing the heart rate and the electrical resistance. In addition, by analyzing acceleration, the photographer, for example, is walking slowly, walking fast, stopping, getting on a vehicle such as a bus or train, Action information can be obtained.

  Then, as shown in FIG. 2B, the extracted biological information and behavior information are added to the corresponding image data as data indicating the “user state”. In this way, data indicating the user status is added to all the image data transferred from the memory card 19 to the hard disk drive 64. Note that the photographer's biometric information and action information are meta-information related to the corresponding image as well as the shooting date and time, shooting position, and shooting conditions.

[2-2-2] Image display example (1)
The image is displayed in the form corresponding to the biological information, the behavior information, or the user state for each image according to the analysis result of [2-2-1].

  FIG. 4 shows the display example. Since seven pieces of image data have been transferred from the memory card 19 to the hard disk drive 64, respective thumbnails are formed from the seven pieces of image data, and the thumbnails are displayed. Is displayed on the display 72.

  In this example, each thumbnail has a different size according to the user status data added to the image data of the corresponding image. That is, when the image of the thumbnail T1 is taken, the user (photographer) is in a normal state and the heart rate is also a normal value, so the thumbnail T1 is displayed in a standard size based on the biological information. Yes. However, when the image of the thumbnail T2 is taken, the user is impressed and the heart rate is high, so the thumbnail T2 is displayed large. The other icons are also the same, and are displayed in a size according to the user state when each image is taken.

  Furthermore, when the thumbnail T3 image was taken, the heart rate was higher than the preset value, so the thumbnail T3 was displayed in a state of fine vibration, a size that periodically changed, or a blinking state. ing.

  In this way, the display form of the thumbnail of the image is changed corresponding to the biological information, behavior information, or user state when the image is taken. Then, when the thumbnail of the target image is double-clicked with the mouse, for example, the target image is displayed on the display 72 in the original size.

[2-2-3] Image display example (2)
As described above, the display of the image is displayed in the form corresponding to the user state for each image according to the analysis result of [2-2-1]. However, in the display method shown in FIG. This is a case where an image, its shooting position, and a user state for the image are linked.

  That is, FIG. 5 shows a case where display is performed on the display 81 of a mobile phone or mobile device, for example. At this time, the mobile phone mobile device has the same function as the digital still camera 10 and the display of the personal computer 60. It has the same function as the function.

  In FIG. 5, image data for six images is transferred, and the display 81 is a vertically long display screen. A map is displayed on the upper half of the display 81, marks M1 to M6 indicating the shooting positions are displayed at the positions where the six images are shot, and the sizes of the marks M1 to M6 are the same. , [2-2-2].

  The display positions of the marks M1 to M6 on the map are determined according to position information added to the image data of the corresponding image. That is, a map including the position indicated by the position information is displayed on the display 81, and marks M1 to M6 are displayed on the map. At this time, for example, when the photographers of the images corresponding to the marks M1 and M4 are the same, the marks M1 and M4 are the same color, for example, red, and the photographer of the image corresponding to the mark M2 corresponds to the marks M1 and M4. When different from the photographer of the image, the mark M2 is set to another color, for example, blue. That is, the colors of the marks M1 to M6 are the same color or different colors corresponding to the photographer of the image corresponding to the mark.

  In the lower half of the display 81, thumbnails T1, T2,... Of captured images are displayed in one horizontal row. In FIG. 5, four thumbnails T1 to T4 are displayed from the beginning, and the remaining two thumbnails are displayed by instructing horizontal scrolling. In this case, the thumbnails T1 to T6 correspond to the marks M1 to M6.

  Further, buttons B1 to B4 for changing the scale of the map shown in the upper half of the screen are displayed at the center of the display 81. In addition, a component box CB is prepared on the upper side of these scale buttons B1 to B4, for example, where the latest upload time of each user, his position, and the place where each user took the latest image. Distance and are displayed.

  The result of selecting the user in this component box is reflected in the upper half map and the lower half thumbnail. In addition, icons IA to IN described later are displayed at the bottom of the display 81.

  Therefore, the image, its shooting position, and the user state for the image are associated with each other. Then, when the mark or thumbnail of the target image is double-clicked with the mouse, for example, the target image is displayed on the display 81 in the original size.

[3] Sharing of Image File As described above, the photographer's biometric information and action information are meta-information related to the corresponding image as well as the shooting date and time, shooting position, and shooting condition. Therefore, if this meta information is used or used, it becomes possible to share an image and it can be used conveniently among image sharers.

  Therefore, the example shown in FIG. 6 is a case where an image is shared. That is, in this case, the user A has the digital still camera 10A and the personal computer 60A, the user B has the digital still camera 10B and the personal computer 60B, ..., the user N has the mobile phone 80N. It is a case.

  The digital still cameras 10A, 10B,... Are configured in the same manner as the digital still camera 10, and the personal computers 60A, 60B,. The mobile phone 80N has the same functions as the display functions of the digital still camera 10 and the personal computer 60. For example, the mobile phone 80N has a display 81 as shown in FIG. Further, map data for displaying a map is prepared in the server 91 as shown in FIG. 5, for example.

  The digital still cameras 10A, 10B,..., The personal computers 60A, 60B,..., And the cellular phone 80N are connected to the server 91 through a network, for example, the Internet 90, by the communication circuits 25 and 67 provided therein. Is done.

  Therefore, the digital still cameras 10A, 10B,... Or an image (and its meta information) taken by the mobile phone 80N can be uploaded to the server 91, and those images (and the meta information) can be uploaded to the personal computer 60A, 60B,... Or can be downloaded to the mobile phone 80N. That is, the images taken by the users A to N can be shared together with the meta information.

  In this case, it is possible to allow only some images to be downloaded to some users, and allow only some images and meta information to be uploaded to other users.

  When sharing an image (and meta information) in this way, as shown in FIG. 5, for example, a photographer of an image in which marks M1 to M6 are displayed in the upper half at the bottom of the display 81. .. Are displayed, and the colors of the marks M1 to M6 are the same as the corresponding icons IA, IB,.

  For example, when the images of the thumbnails T1 and T4 correspond to the marks M1 and M4 and the photographer of the images is indicated by the icon IA, the marks M1 and M4 and the icon IA are red, for example, and the image of the thumbnail T2 Corresponds to the mark M2, and the photographer of the image is indicated by the icon IB, the mark M2 and the icon IB are blue, for example.

  Therefore, the photographer can be identified from the colors of the marks M1 to M6 and the icons IA, IB,.

[4] Summary According to the above-described apparatus, the photographer's biometric information and behavior information are added to the photographed image as more sensitive data. It can be displayed separately from the image. Therefore, for example, when searching for an image, a target image can be easily found.

  In addition, there is a psychological change immediately before pressing the shutter button, and this change may trigger the shutter button, but biometric information and behavior information before and after the shutter button is pressed are added to the image. Therefore, the distinction from other images can be made more clearly. That is, since biometric information and behavioral information up to immediately before photographing of the user (photographer) is added to the image, by estimating from the biometric information and behavioral information, an image in a state different from normal can be searched, You can choose.

[5] Others In the above description, the digital still camera 10 that captures an image of a subject and the personal computer 60 that displays the captured image are separate, but the display function of the personal computer 60 is included in the digital still camera 10. A similar display function can be provided.

  For example, as shown in FIG. 7A, image data is added to image data in the same manner as a general digital still camera, and link information is added, and the image data is designated by the link information as shown in FIG. 7B. You can also include data such as user status in another file. In this way, image data and user status data can be managed independently.

  Furthermore, body temperature, respiratory rate, electrocardiogram data, and the like can be used as biological information. In the above description, when the user presses the shutter button, shooting is performed and data indicating the user state is stored as meta information at that time, but shooting is automatically performed every predetermined period. In addition, data indicating the user state can be stored.

  Moreover, the frequency table of the place where the user behaves is created based on the output of the GPS unit 34 to grasp the range in which the user usually behaves. Can also be saved.

  Further, instead of obtaining the position information by the GPS unit 34, the position information can also be obtained from a base station of a cellular phone or a wireless LAN access point. Further, for example, the digital still camera 10 is suspended near the passenger seat of an automobile, and the traveling direction is automatically photographed, for example, every 10 seconds, and the user state of the driver at this time is similarly recorded to be a drive recorder. You can also.

  In addition, the analysis of the biological information and the behavior information in [2-2-1] can be performed by software or dedicated hardware in the digital still camera 10. Further, the digital still camera 10 or the personal computer 60 can perform image recognition and add the recognition result to the image data.

  Furthermore, the sensors 31 and 32 can be a necklace type, a bracelet type, an armband type, or the like, in addition to the above-described watch type (list type), and can perform analysis in addition to data detection. Separately from imaging, biometric information can also be recorded at regular intervals. Further, the meta information can be a DCF standard or the like.

[List of abbreviations]
A / D: Analog to Digital
AGC: Automatic Gain Control
CCD: Charge Coupled Device
CDS: Correlated Double Sampling
CPU: Central Processing Unit
D / A: Digital to Analog
DCF: Design rule for Camera File system
Exif: EXchangeable Image File format
GPS: Global Positioning System
JPEG: Joint Photographic Experts Group
MP3: MPEG-1 / Audio Layer 3
MPEG: Motion Picture Experts Group
OS: Operating System
RAM: Random Access Memory
ROM: Read Only Memory

It is a systematic diagram showing one embodiment of the present invention. It is a figure which shows an example of the data structure in this invention. It is a systematic diagram showing one embodiment of the present invention. It is a figure which shows the example of a display by this invention. It is a figure which shows the other example of a display by this invention. It is a systematic diagram which shows the usage example of this invention. It is a figure which shows the other example of the data structure in this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Digital still camera, 12 ... CCD image pick-up element, 16 ... Display control circuit, 17 ... Viewfinder, 18 ... Recording / reproducing circuit, 19 ... Memory card, 20 ... Microcomputer, 27 ... Operation button, 31 ... Biological information sensor, 32 ... Behavior information sensor, 33 ... Clock circuit, 34 ... GPS unit

Claims (8)

In a shooting device that takes a picture of a subject and obtains image data,
A biometric information sensor that detects biometric information of the photographer at the time of photographing the subject;
A behavior information sensor for detecting the photographer's behavior information at the time of photographing,
At the time of shooting, the biological information detected by the biological information sensor and the behavior information detected by the behavior information sensor are stored in association with the image data as meta information of the photographer at the time of shooting ,
Having a display means;
While displaying the thumbnail of the photographed image on the display means,
Depending on the biological information and the behavior information, the thumbnail display form is changed.
Image taking device. The imaging device according to claim 1,
At the time of shooting, it has position information detection means for detecting the position information of the shooting location,
An image photographing apparatus in which the position information obtained from the position information detecting means is also stored in association with the image data as part of the meta information. The imaging device according to claim 2,
At the time of shooting, it has a clock circuit for obtaining information on the shooting date and time,
Information of the photographing date and time obtained from the clock circuit also photographing equipment of images to be stored in association with the upper Symbol image data as part of the meta information. In the imaging device according to claim 1, claim 2 or claim 3,
An image photographing apparatus in which the meta information is integrated and stored as part of the image data. The biometric information of the photographer detected by the biometric information sensor at the time of photographing the subject and the action information of the photographer detected by the behavior information sensor at the time of photographing are the meta information of the photographer at the time of photographing. If the image data obtained by shooting is stored in association with it,
In addition to displaying thumbnails of the captured images on the display means,
An image display device in which a display form of the thumbnail is changed according to the biological information and the behavior information. The display device according to claim 5 ,
At the time of shooting, when the location information of the shooting location detected by the location information detection means is also stored in association with the image data as part of the meta information,
A map including the shooting location indicated by the position information is displayed on the display means,
While displaying a predetermined mark at the position indicated by the position information on the displayed map,
An image display device in which the display form of the displayed mark is made different according to the biological information and the behavior information associated with the image data corresponding to the mark. The display device according to claim 5 or 6 ,
An image display device in which the meta information is integrated as part of the image data. The biometric information of the photographer detected by the biometric information sensor at the time of photographing the subject and the action information of the photographer detected by the behavior information sensor at the time of photographing are the meta information of the photographer at the time of photographing. In association with the image data obtained by shooting, together with the image data, upload to the server and share the image data and its meta information ,
The thumbnail of the photographed image is displayed in a display form according to the biological information and the behavior information.
Management system of the image data to.