JP2023162709A - Imaging device - Google Patents

Abstract

To provide an imaging device configured to add and record additional information matched with audio information of a recorded video to the record video using a method according to the additional information.SOLUTION: Included are: imaging means; a microphone; recording means for recording video data from the imaging means and audio data from the microphone on a recording medium; main scene determination means for determining main scenes in the video data recorded on the recording medium; audio analysis means for analyzing the audio data corresponding to the main scenes and extracting a keyword; obtaining means for obtaining a still image or video related to the keyword as a related image; and editing means for editing the recorded video data so as to edit the scene from which the keyword is extracted from among the main scenes. The editing means adds the related image obtained by the obtaining means to the scene from which the keyword is extracted in the video data, using a method according to the related image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device.

2. Description of the Related Art In recent years, cameras that regularly and continuously take pictures without the user giving a shooting instruction have been developed and are being put into practical use. For example, such a camera can be installed anywhere indoors and automatically take pictures to record casual scenes from everyday life.

When playing back moving images and audio recorded with such a camera, simply playing back the moving image as it was recorded may result in a moving image with little movement, resulting in a moving image that is uninteresting for viewers. Therefore, it is desirable to edit the recorded moving image to make it more suitable for viewers and to play it back.

On the other hand, Patent Document 1 discloses a technique of extracting a keyword from input voice, searching for an image corresponding to the keyword from among a plurality of predetermined images, and displaying the image.

Japanese Patent Application Publication No. 2017-16296

In the conventional technology disclosed in Patent Document 1, it is possible to display related images according to audio. However, when similar techniques are applied to audio accompanying recorded moving images, the problem becomes how to add and reproduce related images during playback of the moving image.

Therefore, an object of the present invention is to record additional information that matches the audio recorded together with the moving image by adding it to the moving image using a method that corresponds to the additional information.

The present invention includes an imaging means, a microphone, a recording means for recording moving image data from the imaging means and audio data from the microphone on a recording medium, and a recording means for recording main scenes in the moving image data recorded on the recording medium. a main scene determination means for determining a main scene; an audio analysis means for analyzing audio data corresponding to the main scene and extracting a keyword; an acquisition means for obtaining a still image or a moving image related to the keyword as a related image; The editing means edits the recorded video data so as to edit the scene from which the keyword is extracted, and the editing means edits the related image acquired by the acquisition means according to the related image. The keywords in the video data are added to the extracted scenes in a method according to the method described above.

According to the present invention, additional information matching the audio recorded together with the moving image is added to the moving image and recorded using a method according to the additional information.

FIG. 1 is a block diagram showing an imaging device. It is a flowchart regarding video shooting processing. It is a flowchart regarding video editing processing. It is a figure showing an example of a photographed video and a video after editing.

Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings.

Hereinafter, with reference to FIG. 1, an imaging apparatus 100 in an embodiment of the present invention will be described. A lens 1001 is a lens group including a zoom lens and a focus lens. The lens control unit 1002 has a function of lens control processing that controls the focal length and aperture state of the lens 1001 based on calculation results from an evaluation unit 1010 (described later) and subject information extracted by a recognition unit 1011. The CPU 1003 controls the entire imaging apparatus 100. A CPU bus 1004 performs communication between the CPU 1003 and each functional block.

The RAM control unit 1007 controls access to the RAM 1006 based on RAM access requests from each functional block. The RAM bus 1005 performs communication between the RAM control unit 1007 and each functional block. The RAM bus 1005 also has a function of arbitrating access to the RAM 1006 from each functional block.

The imaging unit 1008 is an imaging unit that converts an optical signal captured by the lens 1001 into an electrical signal using an image sensor (not shown), and performs processing to correct lens aberrations for the obtained image data, and corrects defective pixels of the image sensor. It has a processing function that interpolates. The developing unit 1009 performs debayer processing on the image data generated by the imaging unit 1008 to convert it into a signal consisting of a luminance signal and a color difference signal, removes noise contained in each signal, corrects optical distortion, and processes the image. It has the function of development processing such as optimization of The evaluation unit 1010 has a function of an evaluation value calculation process that calculates evaluation values such as focus state and exposure state based on the image data generated by the imaging unit 1008.

The recognition unit 1011 has a recognition processing function of detecting and recognizing subject information in the image data developed by the developing unit 1009 and generating subject information. For example, if a face is detected in image data, information indicating the position of the face is output, and a specific person is authenticated based on characteristic information such as the face.

The display control unit 1012 has a display control function of performing predetermined display processing on the image data developed by the developing unit 1009 and then outputting the result to the display unit 1015. The display unit 1015 is a reproduction means for reproducing image data, and is configured by, for example, a liquid crystal display.

The video encoding unit 1013 has a video encoding function that compresses and encodes the image data developed by the developing unit 1009 using a predetermined video compression method such as MEPG, and converts it into a video with a compressed amount of information. have The recording control unit 1014 has a function of recording control processing to record the image data developed by the developing unit 1009 onto the recording medium 1016. The recording medium 1016 is, for example, a nonvolatile memory card or a hard disk.

Microphone 1017 converts voice into an audio signal. The microphone control unit 1018 is connected to the microphone 1017 and has functions such as controlling the microphone 1017, starting and stopping sound collection, and acquiring audio from the microphone 1017. Control of the microphone 1017 includes, for example, gain adjustment and status acquisition. Audio encoding/decoding section 1019 encodes or decodes the audio signal from microphone 1017 using a predetermined encoding method such as AAC. Speaker 1020 outputs the audio signal decoded by audio encoder/decoder 1019.

The voice analysis unit 1021 is a voice analysis unit that analyzes the voice acquired by the microphone 1017, and performs analysis such as detecting a specific keyword in the voice data and detecting a change in volume in the voice data.

The communication unit 1022 is a communication interface that connects the imaging device 100 and other devices by wire or wirelessly, and transmits and receives image data, audio data, etc., and can also be connected to a network such as a wireless LAN or the Internet. The communication unit 1022 can transmit image data acquired by the imaging device 100 and image data recorded in the recording unit 1016 to the outside, and can receive image data and various information from external devices.

The operation unit 1023 accepts various operations from the user to perform various settings of the imaging device 100. The related image acquisition unit 1024 acquires related images related to the keyword detected by the voice analysis unit 1021 from the Internet connected via the recording unit 1016 or the communication unit 1022.

The main scene determination unit 1025 is a main scene determination unit that determines whether the scene being photographed is a main scene based on the image recognition result by the recognition unit 1011 or the audio analysis result by the audio analysis unit 1021. The video editing unit 1026 is a video editing unit that edits the video recorded on the recording medium 1016 and adds related images acquired by the related image acquisition unit 1024.

Next, with reference to FIG. 2, a video capturing process in the imaging device 100 will be described. The flowchart in FIG. 2 is executed when the power switch (not shown) of the imaging device 100 is turned on and set to video shooting mode.

In step S201, video shooting is started by the user's operation on the operation unit 1023 to instruct the start of shooting. In step S202, the CPU 1003 starts video recording processing and audio acquisition processing. In the video recording process, the image capturing unit 1008 continuously captures images at a set frame rate, the acquired image signal is encoded by the video encoding unit 1013, and is sent as video data via the RAM bus 1005 and the RAM control unit 1007. and records it in the RAM 1006.

In addition, in parallel with recording the video, the microphone control unit 1018 acquires audio from the microphone 1017, the audio encoder/decoder 1019 encodes the acquired audio data, and the audio data accompanying the video data is stored in the RAM 1006. to be recorded. The moving image data and audio data recorded in the RAM 1006 are recorded on the recording medium 1016 via the recording control unit 1014 as one moving image file. This series of video recording processing and audio acquisition processing is repeatedly executed until the user performs an operation to instruct the end of shooting.

In step S203, it is determined whether the user operates the operation unit 1023 to instruct the end of photographing. If there is an operation to instruct the end of photographing, the process advances to step S208. If there is no operation to instruct the end of photographing, the process advances to step S204.

In step S204, the main scene determination unit 1025 determines whether the scene being photographed is a main scene. The main scene is, for example, when the volume of the scene being shot increases. If it is determined that the scene being photographed is a main scene, the process advances to step S205. If it is determined that the scene is not a main scene, the process returns to step S203.

In step S205, the audio analysis unit 1021 analyzes audio data from the microphone 1017. In step S206, the voice analysis unit 1021 determines whether the voice data includes a keyword. It is assumed that the keyword is set by the user in advance and recorded in the recording unit 1016. Further, it is assumed that the keyword is, for example, a place name indicating a specific place or a person name indicating a specific person. If the voice data includes a keyword, the process advances to step S207. If there is no keyword, the process returns to step S203.

In step S207, the video encoding unit 1013 adds an internal marker to a scene that is determined to have a keyword among the acquired videos, in conjunction with video data encoding processing. The information given as an internal marker includes information indicating a keyword acquired within the scene. The assigned internal marker serves as a mark of an editing point in video editing processing, which will be described later.

In step S208, the moving image and audio continuously recorded by the moving image recording process from the start to the end of shooting the moving image are recorded on the recording medium 1016 as one moving image file, and the moving image file is closed. Then, the series of processing ends.

In this way, when a video is shot, if a keyword is included in the audio to be recorded, an internal marker that serves as a mark of an editing point is added to a scene in the corresponding video data.

Next, editing processing for recorded video files will be described with reference to FIG. The flowchart in FIG. 3 is continuously and automatically executed by the CPU 1003 after the video shooting process in FIG. 2 is completed.

In step S301, it is determined whether or not there is an internal marker in the video file recorded on the recording medium 1016 by the CPU 1003 that has not been edited yet. If there is an unprocessed internal marker, the process advances to step S302. If there are no unprocessed internal markers, the process ends.

In step S302, keywords are acquired from unprocessed internal markers. In step S303, the related image acquisition unit 1024 acquires a related image related to the keyword acquired in step S302. For example, if the keyword is a place name indicating a specific place, the related image is an image taken at the location indicated by the keyword. Furthermore, when the keyword indicates a specific person, the image is set to include the person indicated by the keyword as a subject. The related image acquisition unit 1024 obtains related images from other video files recorded on the recording medium 1016. Further, the related image acquisition unit 1024 acquires related images from a server or the like on a network (not shown) connected via the communication unit 1022.

In step S304, it is determined whether the related image acquired in step S303 is a moving image. If it is a video, the process advances to step 305. If the image is a still image rather than a moving image, the process advances to step S307.

In step S305, it is determined whether the video acquired in step S303 has a length longer than a predetermined time. If the video is longer than the predetermined time, the process advances to step S306. If the video is shorter than the predetermined time, the video is treated as a related video and the process advances to S307.

In step S306, a predetermined period of time is cut out from the video acquired in step S303 to create a related video for editing. The video may be cut out, for example, by cutting out a predetermined amount of time from the beginning of the video, or by cutting out a predetermined amount of time after a certain amount of time has elapsed from the beginning.

In step S307, the video editing unit 1026 adds the related image acquired in step S303 or the related video created in step S306 to the scene to which the internal marker of the video file recorded in the recording unit 1016 is attached. Make edits.

If the related image acquired in step S303 is a still image, the video file is edited so that the still image is displayed for a predetermined period of time at the timing of a scene to which an internal marker is attached in the video file. If the related image acquired in step S303 is a video, the video or the related video cut out in step S306 is edited so as to be played back at the timing of the scene to which the internal marker is attached in the video file.

The editing method here is determined depending on the content of the still image or moving image that is a related image to be added to the moving image file. For example, if the main subject is included in a related image, there is a high possibility that the related image is of high importance to the viewer, so the video file is created by inserting the related image into the relevant scene for a predetermined period of time and displaying it. Edit. If the main subject is not included in the related image, it is unlikely that the related image is of high importance to the viewer, so edit the video file by displaying it in a small superimposed position, such as in the lower right corner of the relevant scene. do.

After step S307, the process returns to step S301 and repeats a series of processes until the video editing process is completed for all internal markers added to the video file.

In this way, by acquiring images related to the keywords detected by the audio analysis unit 1021 and creating a video with the related images added, a video more suitable for viewers can be obtained.

FIG. 4 shows an example of a moving image captured by the imaging device 100 and a moving image after related images are added to the moving image. FIG. 4(a) is a moving image captured by the imaging device 100. The moving image shown in FIG. 4(a) includes scenes 401 to 403 along the flow of time. When the audio analysis unit 1021 detects a keyword in the audio data accompanying the scene 401, a related image is added after the scene 401, and a moving image as shown in FIG. 4(b) is created. In FIG. 4B, a related image 404 is inserted between a scene 401 and a scene 402.

As described above, the present embodiment aims to provide an imaging device that adds and records additional information matching audio information of a recorded video to a recorded video using a method according to the additional information. .

In the above-described embodiment, the related image acquisition unit 1024 has acquired one related image, but when multiple images are acquired, the images with the latest shooting date and time are prioritized among the acquired images. It may be added.

Further, although the main scene in step S204 of FIG. 2 is defined as a time when the volume of audio data of the scene being photographed increases, the main scene is not limited to this. For example, it may be set when multiple subjects in the scene are talking, or when there is a large change in emotion indicated by the audio data.

Furthermore, in order to improve the viewing experience of the edited video file, the number of times a recorded video file can be edited may be limited to once or less every predetermined period of time.

Furthermore, in order to improve the viewing experience, a video file that has been edited many times may be suggested to the user as a recommended video so that video files that have been edited many times are played preferentially over video files that have been edited few times.

Further, although the video editing process is started after the video shooting ends, the video editing process may be started in response to a start instruction from the user. The user's start instruction may be, for example, an operation on the operation unit 1023 or a voice instruction via the microphone 1017. Although the display unit 1015 is, for example, a liquid crystal display, it is not limited to this. For example, it may have a projector function and project images outside the imaging device 100.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the invention.

100 Imaging device

Claims (14)

an imaging means;
Mike and
recording means for recording video data from the imaging means and audio data from the microphone on a recording medium;
Main scene determining means for determining a main scene in video data recorded on the recording medium;
audio analysis means for analyzing audio data corresponding to the main scenes and extracting keywords;
acquisition means for acquiring a still image or video related to the keyword as a related image;
comprising editing means for editing the recorded video data so as to edit a scene from which the keyword is extracted from among the main scenes;
The image capturing apparatus is characterized in that the editing means adds the related image acquired by the acquisition means to the scene from which the keyword is extracted in the moving image data in a method according to the related image. The imaging device according to claim 1, wherein the acquisition means acquires another image recorded on the recording medium or the related image from a network. When the related image acquired by the acquisition means is a video, the editing means creates a cutout video cut out from the related image for a predetermined period of time, and adds the cutout video to the video data. , The imaging device according to claim 1 or 2. Claims 1 to 3, wherein when the related image includes a main subject, the editing means inserts the related image into a scene from which the keyword is extracted in the video data. The imaging device according to any one of the above. If the main subject is not included in the related image, the editing means superimposes the related image in a small size on the scene from which the keyword is extracted in the video data. 4. The imaging device according to any one of 4. When there are a plurality of related images acquired by the acquisition means, the video editing means adds the image to the video data preferentially starting from the image with the latest shooting date and time among the plurality of related images. The imaging device according to any one of claims 1 to 5. The imaging device according to any one of claims 1 to 6, wherein the main scene is a scene in which the volume of audio corresponding to the video data is higher than a predetermined value. The imaging device according to any one of claims 1 to 6, wherein the main scene is a scene in which a plurality of subjects in the video data are talking. The imaging device according to claim 1, wherein the main scene is a scene in which the emotion of audio in the video data changes significantly. 10. The imaging device according to claim 1, wherein when the keyword represents a place name, the acquisition means acquires still images and moving images taken at the location indicated by the keyword. The imaging device according to any one of claims 1 to 9, wherein when the keyword represents a person's name, the acquisition means acquires still images and videos that include the person indicated by the keyword as a subject. . 12. The imaging device according to claim 1, wherein the editing unit edits the video data no more than once every predetermined time period of the video data. The editing means records the video data edited by the video editing means on the recording means,
13. Any one of claims 1 to 12, further comprising a reproducing means for reproducing, as a recommended moving image, the moving image data having more scenes edited by the editing means from among the moving image data recorded on the recording medium. The imaging device described in section. The imaging device according to claim 1, wherein the editing means edits the video data according to instructions from a user.

Images