JP2010237761A - Electronic apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately evaluate the content of an image, music or the like. <P>SOLUTION: When an image is reproduced as content, the face of an observer is imaged by a sub-camera, and the voice of the observer is recorded. On the basis of the acquired face image data, expression recognition and body motion recognition are performed, and scoring is performed based on the result. On the other hand, voice recognition is performed based on the recorded voice of the observer, and scoring is performed based on the result. The evaluation of the image is determined based on the total value of the results of scoring. The determined evaluation is stored in association with the image data. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an electronic device that can automatically evaluate content such as images and music.

  It is possible to detect the viewer's body movement while playing content such as music and video, calculate the viewer's preference for the content, and record the preference in association with the song A reproducing apparatus is known (see, for example, Patent Document 1).

JP 2007-86835 A

  However, it is often difficult to determine a viewer's preference level based on body movement.

  The electronic apparatus according to the present invention includes a reproducing unit that reproduces content, a detecting unit that detects a viewer's facial expression and / or audio during reproduction of the content, and the reproduced content based on a detection result of the detecting unit. And an evaluation means for evaluating.

  According to the present invention, since the content is evaluated by the viewer's facial expression and / or voice, the content can be accurately evaluated.

The control block diagram of the digital camera in one Embodiment. The figure which shows the example of a display at the time of displaying a rating result superimposed on an image. The figure which shows the example of a display at the time of overlaying and displaying the state of the viewer at the time of rating on an image. The flowchart explaining the reproduction | regeneration rating function at the time of a slide show. The figure which shows the example of a display at the time of carrying out the graph display of the rating result after a slide show.

An embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram showing a camera control system in the present embodiment. An imaging light beam that has passed through an imaging lens (not shown) is imaged by an imaging sensor 2 through an optical low-pass filter 1. The operation of the image sensor 2 is controlled by a timing generator (TG) 3.

  An imaging signal that is a photoelectric conversion output of the imaging sensor 2 is converted into a digital signal by the A / D converter 4 and input to the image processing MCU 5, and processed by a DSP (digital signal processing device) 5 a to generate image data. . The image data is recorded on an image recording medium such as the memory card 6 as an image file.

  In the reproduction mode, an image file recorded on the memory card 6 or the like is read out, and after being processed by the DSP 5a and the display output unit 5b, can be displayed as an image on the built-in LCD unit 7. The LCD unit 7 is a color liquid crystal monitor provided on the back of the camera, for example. In the playback mode, a slide show can be created and executed. The slide show is a function for displaying a plurality of image data in the memory card 6 on the LCD unit 7 by sequentially exchanging them at a predetermined time interval.

  Furthermore, the image data can be converted into a video signal and displayed on an external device such as a TV via the video output unit 8. Further, the image file can be output to an external high-definition TV or the like via the high-resolution output transmitter 9.

  A microphone 10, a speaker 11, and a sub camera 12 are connected to the image processing MCU 5, and audio data input through the microphone 10 can be recorded and recorded audio data can be reproduced from the speaker 11. The sub camera 12 has a dedicated photographing lens, an image sensor, and the like, and is provided at a position where the face of the viewer who appreciates the image on the LCD unit 7 can be imaged. For example, when the LCD unit 7 is provided on the back side of the camera, the photographing lens of the sub camera 12 is also provided on the back side of the camera.

  The sub MCU 20 connected to the image processing MCU 5 responds to the operation of the release button 21 and other operation unit 22 and receives input information from the sensor (AF sensor, AE sensor, etc.) 22A and the posture detection unit 23. Various calculations are performed based on camera posture information and the like, and the camera drive unit 24 is controlled based on the calculation results. The camera driving unit 24 includes an AF driving unit (lens driving unit), an aperture driving unit, a mirror driving unit, and the like.

Next, the image rating function will be described.
The camera according to the present embodiment has a function of detecting the facial expression and utterance of the viewer during image reproduction and automatically evaluating the image based on the detection results. Hereinafter, this function is referred to as a “reproduction rating function”.

  In order to realize this function, the camera has a face recognition function, a facial expression recognition function, and a voice recognition function. Face recognition is a function for detecting the position and size of a person's face area when there is a person as a subject in image data as is well known, and generally focuses and exposes the face portion. Used for Facial expression recognition is a function that recognizes facial expressions in image data recognized by face recognition. For example, facial expressions such as “smile”, “joy”, “sadness”, “anger”, “surprise” Can be detected. This can be realized by using a technique described in, for example, Japanese Patent Application Laid-Open No. 2001-51338. Speech recognition is a function for recognizing speech content from speech data as is well known.

  The playback rating function can specify selection / non-selection on a setting screen or the like. If an image is reproduced in the reproduction mode when this function is selected, the sub camera 12 and the microphone 10 are operated to perform imaging and recording, respectively. As described above, the sub camera 12 captures the face of the viewer of the image and generates image data.

  The image processing MCU 5 performs face recognition on the face image data obtained by the sub camera 12, and then performs facial expression recognition. In addition, voice recognition is performed on the recorded voice data to acquire the contents of the voice and to detect the volume and tone of the voice. Then, the reproduced image is evaluated based on the results of facial expression recognition and voice recognition.

  For example, when an expression of “smile”, “joy”, or “surprise” is detected, the score is added, and when “sadness”, “anger”, etc., the score is subtracted. If no emotional expression is detected, score addition / subtraction is not performed. In addition, when a plurality of people appreciate an image, a plurality of faces are recognized. In this case, the score is determined based on the expression recognition result for the plurality of faces. For example, the score is obtained for each face, and the maximum value or average value thereof is obtained. The score may be changed based on the degree of facial expression (for example, even if the same “smile” is used, the score is changed depending on the degree of smile and big smile).

  For voice, the score is added / subtracted according to the loudness and tone. For example, it is determined that the greater the voice or the higher the tone, the greater the impression, and the score is added. Further, based on the result of the voice recognition, when an honor word such as “awesome” is detected, a score is added.

  In addition to the facial expressions and voices, scoring may be performed by body movement. The body movement can be determined from the image data repeatedly obtained by the sub-camera 12. For example, the score may be added when the body of the viewer moves greatly or when a gesture of tapping the hand is detected. Furthermore, moving image shooting may be performed by the sub camera 12, and facial expression recognition and body movement recognition may be performed based on the moving image.

  If a face area is not detected in the image of the sub camera 12 for some reason, scoring cannot be performed by facial expression recognition or body motion recognition. In this case, the current scoring may not be performed, or the scoring may be performed only by the audio result. If no voice is detected, the current scoring may not be performed. If a face area is detected, scoring may be performed only by facial expression recognition or body motion recognition. Good.

  The series of processes from the imaging and recording of the viewer to scoring may be performed at least once during the reproduction of one image. However, it is repeated at a predetermined cycle during reproduction, and the maximum value of each time or The average value may be the score of the image.

  The score obtained as described above may be used as the evaluation of the image as it is, but an evaluation such as a five-level evaluation or a ten-level evaluation may be performed based on the score.

  The determined evaluation is stored in association with the image. In general, an image is recorded on a recording medium such as a memory card 6 as an image file that can be handled by a computer. The evaluation information can be embedded as header information of the image file. Evaluation data may be stored separately from the image file.

  When the rated image is reproduced and displayed on the built-in LCD unit 7, the evaluation information associated with the image file is read and displayed together with the image. FIG. 2 shows an example of the display, in which the result of the five-level evaluation is displayed superimposed on the image with the number of stars. You may display in the margin part of an image. This rating display can also be displayed on each thumbnail image in the thumbnail display for image selection.

  On the other hand, as shown in FIG. 3, an image used for the rating (a photograph of the viewer, a still image or a moving image) is displayed on the sub-window SW, and the audio used for the rating is reproduced, thereby enabling the viewer. You can enjoy images with the reaction. In order to realize this, it is necessary to store the image data of the viewer and the recorded audio data in association with the image file to be evaluated.

  Here, the rating can also be used as a key for image classification, rearrangement, and image search. For example, the rated image file may be automatically moved to a folder created for each evaluation (for example, a five-star image file is put in a five-star folder). In addition, images are rearranged in the order of evaluation, some images with high (low) evaluation are selected and displayed (for example, top 10 and worst 10), and specific evaluation (for example, 5 stars, 3 stars) It is also easy to search only thumbnail images and display thumbnails.

  Such processing using the rating result can also be executed by an external device such as a personal computer. However, in that case, it is necessary to incorporate dedicated software in the external device and transfer the rated image file.

  Note that when a rating-completed image is reproduced, the rating may not be performed again, or the rating may be performed again to rewrite the result, or the result may be accumulated. Any of the above may be selectable.

  The playback rating function as described above can be performed when the user selects and plays back images one by one or when a slide show is executed.

FIG. 4 is a flowchart showing an example of the playback rating process in the slide show.
When the slide show is started, this program is started. First, in step S1, the number counter N is set to “1”. In step S2, the Nth image is reproduced and displayed, and in step S3, imaging by the sub camera 12 and recording by the microphone 10 are performed. In step S4, face recognition is performed on the image data obtained by the sub camera 12, and it is determined whether or not a face area is detected in step S5. If a face area is detected, facial expression recognition is performed in step S6 and body motion recognition is performed in step S7. In step S8, the pitch and tone of the recorded voice data are detected and voice recognition is performed. In step S9, the Nth display image is scored as described above based on the results of steps S6 to S8.

  In step S10, it is determined whether or not a predetermined time (one display time in the slide show) has elapsed since the start of displaying the Nth image. If not, the process returns to step S3. Proceed to step S11. In step S11, the evaluation of the Nth image is determined based on, for example, the maximum value of each time score. The determined evaluation information is stored in association with the image file.

  In step S12, it is determined whether or not the slide show has ended. If all the predetermined images have been reproduced, or if the user has performed an operation for forcibly terminating the slide show, step S12 is affirmed and the process proceeds to step S13. In step S13, for example, as shown in FIG. 5, the evaluation result of each image is displayed in a graph. In FIG. 5, vertical lines indicate the evaluation of each image in the display order, and thumbnail images are displayed particularly for images whose evaluation is greater than or equal to the reference value. Thereby, a highly evaluated image can be confirmed at a glance.

  If the slide show is not yet finished, the process proceeds to step S14, N is incremented, and the process returns to step S2.

  In view of the fact that the evaluation varies depending on the viewer, the rating may be performed for each viewer. That is, the evaluation by the viewer A, the evaluation by the viewer B, the evaluation by the viewer C,... Can be separately obtained and stored for one image. To realize this, a plurality of person names (A, B, C...) And facial feature data are registered in advance in a database provided in the camera. The facial feature data is automatically acquired by the camera from data obtained by imaging each person's face. After that, when the playback rating function is executed, the camera performs face recognition on the image data from the sub camera 12, and performs the above database search to determine who is the viewer (who and who in the case of multiple users). ) Then, facial expression recognition is performed on each person, and each person's evaluation is obtained and stored based on the result. According to this, it is interesting to know what kind of images each viewer likes.

  In addition, the image obtained by the sub-camera is used for rating. For example, if the camera is a lens-rotating type camera that can direct the taking lens toward the viewer viewing the LCD unit, the main camera function The image obtained in step 1 can be used for rating, and a sub camera is not required. Also, when the playback rating function is activated when an image is displayed on an external device such as a TV, a rating image can be obtained with the main camera function.

  Furthermore, the content to be evaluated is not limited to a still image, and may be a moving image, music, or other content.

2 Imaging sensor 5 Image processing MCU
7 Built-in LCD unit 10 Microphone 12 Sub camera

Claims (10)

Playback means for playing back content;
Detection means for detecting the viewer's facial expression and / or audio during content playback;
An electronic device comprising: an evaluation unit that evaluates the reproduced content based on a detection result of the detection unit. The electronic device according to claim 1, wherein the detection unit includes an imaging unit that images a viewer who appreciates the content, and a facial expression detection unit that detects a facial expression of the viewer from an output of the imaging unit. machine. 3. The electronic apparatus according to claim 1, wherein the reproduction unit is an image reproduction unit that reproduces and displays an image as the content on a built-in display device or an external display device. The reproduction unit is an image reproduction unit that reproduces and displays an image as the content on a built-in display device, and the imaging unit is provided at a position where a viewer who appreciates the image on the built-in display device can be imaged. The electronic device according to claim 2. The electronic apparatus according to claim 1, further comprising an acquisition unit configured to acquire audio content from the detected audio, wherein the evaluation unit evaluates the reproduced content based on the audio content. . The electronic apparatus according to claim 1, further comprising an imaging unit that captures an image of a subject and obtains an image as the content. The electronic apparatus according to claim 1, further comprising a notification unit that notifies the evaluation of the content when the content is reproduced after the evaluation by the evaluation unit. The electronic apparatus according to claim 2, wherein when the content is reproduced after the evaluation by the evaluation unit, the output of the imaging unit is displayed as an image. The electronic device according to claim 1, wherein the evaluation result of the evaluation unit is stored in association with the content. The electronic device according to any one of claims 1 to 9, wherein the detection unit detects a body movement of the viewer in addition to the facial expression and / or voice of the viewer.

Images