JP2019075638A - Image processing device - Google Patents

Abstract

To provide an image processing device capable of automatically extracting highlight scene when reproducing/editing video data after shooting a video.SOLUTION: The image processing device for processing image signals obtained by shooting a subject, comprises: a video processing part configured to generate and record a series of motion images; an information acquisition part configured to acquire a piece of release information from an image processing terminal other than the image processing device; and a highlight extraction part configured to stores the release information and to extract highlight scenes in the motion images generated by the video processing part on the basis of the stored information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus that automatically extracts a highlight scene.

  2. Description of the Related Art Conventionally, as an image processing apparatus, an imaging apparatus is known which captures an object, generates moving image data based on the obtained image, and records the data on a recording medium.

  In recent years, in such a digital camera, as the capacity of a flash memory has been increased, the speed of image processing has been increased and the power consumption has been reduced, and a large amount of photographing has become possible. Patent Document 1 proposes a technique for extracting and reproducing only a scene having a large volume from moving pictures such as a television program.

JP 2007-267351 A

  However, in the method of Patent Document 1, since the highlight scene is extracted by the volume, the highlight scene can not be extracted in a scene where there is not much change in volume. In addition, there is a possibility of erroneous extraction even for sudden volume change that is not a highlight scene.

  An object of the present invention is to provide an image processing apparatus capable of automatically extracting a highlight scene without depending on a change in the volume of the surroundings when viewing and editing captured moving image data.

In order to achieve the above object, an image processing apparatus according to the present invention is
An image processing apparatus for imaging a subject and processing an obtained image signal, comprising:
A moving image processing unit that generates and records a series of moving images;
An information acquisition unit that acquires release information from an image processing terminal other than the image processing apparatus;
And a highlight extraction unit that accumulates the release information and extracts a highlight scene of a moving image generated by the moving image processing unit based on the accumulated information.

  According to the image processing apparatus of the present invention, it is possible to easily extract an editing point by using the provided highlight information when viewing and editing moving image data after moving image shooting.

It is a figure showing composition of an imaging device. It is a flowchart which shows the moving image imaging | photography process in 1st Embodiment. It is a figure showing communication processing concerning a 1st embodiment. It is a figure which shows highlight scene detection in this embodiment. It is a flowchart which shows the process in 2nd Embodiment.

  Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to the following embodiments. The following embodiments do not limit the invention according to the claims, and are not necessarily essential to the solution means of all the inventions of the combinations of the features described in the embodiments.

  The functional blocks described in the present embodiment do not necessarily have to be individual hardware. That is, for example, the functions of several functional blocks may be performed by one hardware. Also, the function of one functional block or the functions of multiple functional blocks may be performed by the coordinated operation of several hardware. Also, the function of each functional block may be executed by a computer program developed by the CPU on the memory.

First Embodiment
<Overall configuration>
First, the configuration of the imaging device 100 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 1, the imaging apparatus 100 according to this embodiment includes a CPU 101, a RAM 102, a ROM 103, and an operation unit 104. The imaging apparatus 100 further includes an imaging unit 110, an image processing unit 111, a microphone unit 120, an audio processing unit 121, and a speaker unit 122. The imaging apparatus 100 further includes an encoding processing unit 130, a display unit 140, a display control unit 141, a recording and reproducing unit 150, a recording medium 151, and a communication unit 160. CPU is an abbreviation for Central Processing Unit. RAM is an abbreviation of Random Access Memory. ROM is an abbreviation of Read Only Memory. In the imaging apparatus 100 of the present embodiment, H.264 is used as a moving image compression method. H.264 and H. Although the H.265 compression scheme is used, any type of compression scheme may be used.

  In the imaging apparatus 100 of the present embodiment, the CPU 101 uses the RAM 102 as a work memory, develops various programs recorded in the ROM 103 in the RAM 102, and controls each block of the imaging apparatus 100 according to the program.

  The operation unit 104 has switches for inputting various operations such as, for example, a power button, a recording button, a zoom adjustment button, an autofocus button, a menu display button, a mode switching switch, and a determination button. In addition, any type of operator such as a cursor key, a pointing device, a touch panel, a dial, etc. may be used. The operation unit 104 transmits an operation signal to the CPU 101 when the user operates these keys and buttons or the touch panel. Various members of the operation unit 104 may be realized as various function icons displayed on the display unit 140. The user can select and operate these function icons. A function is appropriately assigned to the function icon for each scene. Thus, the function icon acts as various function buttons. The function buttons include, for example, an end button, a back button, an image feed button, a jump button, a narrowing button, and an attribute change button. For example, when the menu button is pressed, various settable menu screens are displayed on the display unit 140. The user can intuitively perform various settings using the menu screen displayed on the display unit 140 and the four-direction button or the SET button on the top, bottom, left, and right. The operation unit 104 may be a touch panel capable of detecting a touch on the display unit. The touch panel may be any of various touch panels such as resistive film type, capacitive type, surface acoustic wave type, infrared type, electromagnetic induction type, screen recognition type, light sensor type, etc. .

  The imaging unit 110 controls an amount of light of an optical image of a subject taken in by a lens with an aperture, converts the image into an image signal by an imaging element such as a CCD sensor or a CMOS sensor, and obtains a digital image signal And temporarily stored in the RAM 102. The digital image signal stored in the RAM 102 is then transmitted to the image processing unit 111.

  The image processing unit 111 performs image quality adjustment processing for adjusting the white balance, color, brightness, and the like of the digital image signal based on the set value determined by the user and the set value automatically determined from the characteristics of the image. . Then, the image processing unit 111 stores the processed digital image signal in the RAM 102 again. Further, the digital image signal subjected to the image quality adjustment processing or not processed can be transmitted to the display control unit 141 and displayed on the display unit 140 as an image being captured. In addition, at the time of reproduction, the image processing unit 111 reads out the still image file and the image data included in the moving image file which are read from the recording medium 151 by the recording and reproducing unit 150 and decoded by the encoding processing unit 130. Do. Then, the digital image signal whose image quality has been adjusted or not processed may be transmitted to the display control unit 141 and displayed on the display unit 140 as an image.

  At the time of recording, the encoding processing unit 130 performs image compression processing on the digital image signal processed by the image processing unit 111 and stored in the RAM 102 to generate compressed moving image data and still image data, It is temporarily stored in the RAM 102. Further, at the time of reproduction, processing for decoding digital image signals by decoding compressed moving image data and still image data of an image file read from the recording medium 151 is performed and stored in the RAM 102. In the present embodiment, the encoding processing unit 130 uses H.264 video data. Coding according to the H.264 scheme. The image processing unit 111 also encodes still image data according to the JPEG method. H. In the moving picture coding method such as H.264, data of each frame is coded using intra-frame predictive coding and inter-frame predictive coding. Generally, a frame coded by intraframe predictive coding is called an I picture. Also, image data that has been interframe encoded using a forward frame as a reference frame and a difference with the reference frame (prediction error) is called a P picture. Also, image data that has been interframe encoded using the forward and backward frames as a reference frame and the difference from the reference frame is referred to as a B picture.

  The microphone unit 120 has, for example, an omnidirectional microphone and an AD converter incorporated in the housing of the imaging device 100. In the microphone unit 120, surrounding sound is collected (collected) by the microphone, and the acquired analog sound signal is converted into a digital signal by the AD conversion unit and temporarily stored in the RAM 102. The digital signal stored in the RAM 102 is then transmitted to the audio processing unit 121. At the time of recording, the audio processing unit 121 reads out the digital audio signal stored in the RAM 102, performs processing such as level optimization processing and noise reduction processing, and stores the processed digital audio signal in the RAM 102 again. .

  In addition, the audio processing unit 121 performs processing to compress an audio signal as necessary. Further, at the time of reproduction, the audio processing unit 121 performs processing of decoding compressed audio data included in the audio file and the moving image file read from the recording medium 151 by the recording and reproducing unit 150 and processing of optimizing audio level. . Then, the audio processing unit 121 sequentially stores the processed audio data in the RAM 102.

  The speaker unit 122 has a speaker and a DA converter. In the speaker unit 122, the audio processing unit 121 reads out the digital audio signal stored in the RAM 102, converts it into an analog audio signal, and outputs the audio from the speaker according to the analog audio signal.

  The display unit 140 includes, for example, a liquid crystal display device or an organic EL display device, and displays an image under the control of the display control unit 141. The display unit 140 may be any device such as an LED display as long as it can provide an image to the user.

  The display control unit 141 displays an image on the display unit 140 based on the digital image signal processed by the image processing unit 111 and stored in the RAM 102. The display control unit 141 may also perform image signal processing such as matrix conversion, brightness adjustment, contrast adjustment, gamma adjustment, chroma gain adjustment, and sharpness adjustment on a video signal based on a digital image signal displayed on the display unit 140. Good.

  At the time of moving image recording, the recording / reproducing unit 150 reads moving image data, still image data generated by the encoding processing unit 130 stored in the RAM 102, and audio data generated by the audio processing unit 121. Then, the recording / reproducing unit 150 writes the read moving image data and audio data into the recording medium 151 as a moving image file together with various information such as the shooting date. Further, at the time of still image recording, the recording / reproducing unit 150 records the still image data stored in the RAM 102 as a still image file on the recording medium 151 together with various information such as a shooting date. When recording a moving image file on the recording medium 151, the recording / reproducing unit 150 forms a data stream composed of compressed moving image data and audio data, and sequentially records the data stream on the recording medium 151.

  The recording and reproducing unit 150 also adds management information such as a file header to the data recorded on the recording medium 151. The recording and reproducing unit 150 adds management information such as a file header to the data recorded on the recording medium 151. The recording and reproducing unit 150 manages moving image files and still image files recorded in the recording medium 151 in accordance with a file system such as FAT or exFAT. Further, at the time of reproduction, the recording / reproducing unit 150 reads a moving image file or a still image file recorded in the recording medium 151 according to the file system.

  The CPU 101 analyzes a header (management information) included in the read moving image file and still image file, and extracts compressed moving image data, audio data, and still image data. The extracted compressed moving image data and still image data are stored in the RAM 102 and decoded by the encoding processing unit 130. The compressed audio data is decompressed by the audio processing unit 121. In the present embodiment, moving image data and audio data recorded in the moving image recording mode are recorded as a moving image file of MP4 file format. In the MP4 file, in addition to video and audio data, management information necessary for reproducing the video and audio data is stored in one file.

  Also, the recording medium 151 may be a recording medium built in the imaging device or a removable recording medium. For example, the recording medium 151 includes any recording medium such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a non-volatile semiconductor memory, and a flash memory. When a removable recording medium is used, the recording and reproducing unit 150 includes a mechanism for mounting and ejecting the removable recording medium.

  Further, the communication unit 160 transmits and receives control signals, moving image files, still image files, various data, and the like to and from an external device different from the imaging device 100. The communication method of the communication unit 160 may be in any form, whether wired or wireless.

  Here, each of the image processing unit 111, the audio processing unit 121, the encoding processing unit 130, the display and control unit 141, and the recording and reproducing unit 150 described above is a microcomputer equipped with a program for executing the respective functions described above. May be Also, the CPU 101 may execute at least a part of these functions.

<Basic operation>
Next, the operation of the imaging apparatus 100 according to the present embodiment will be described with reference to FIG.

  In the imaging apparatus 100 of the present embodiment, the CPU 101 powers each block of the imaging apparatus 100 into a power supply unit (not shown) when the user operates the operation unit 104 and inputs an instruction to turn on the power from the operation unit 104. Control to supply

  Next, the CPU 101 determines whether the camera mode set by the operation unit 104 is the reproduction mode, the still image recording mode, or the moving image recording mode (S201). If the moving image mode is not set, the process proceeds to other mode processing (S202).

  When the mode set by the operation unit 104 is a moving image recording mode, the CPU 101 controls each block of the imaging device 100 to execute the following operation.

  First, the imaging unit 110 transmits the obtained digital image signal to the RAM 102 and temporarily stores it. Next, the image processing unit 111 performs the various image quality adjustment processes described above on the digital image signal stored in the RAM 102 in accordance with the set shooting setting, and stores the processed digital image signal in the RAM 102 again. Further, the display control unit 141 reads out the processed digital image signal stored in the RAM 102 or the unprocessed digital image signal and causes the display unit 140 to display the read out digital image signal.

  That is, an image based on the image captured by the imaging unit 110 is displayed on the display unit 140 until the moving image recording start instruction is input from the operation unit 104 (S203).

  Next, in the recording standby state, when an instruction to start moving image recording is input from the operation unit 104, the communication unit 160 performs communication processing with the peripheral image processing terminal (S204).

  Here, communication processing will be described.

  When an instruction to start moving image recording is input, as shown in FIG. 3, communication is started with a camera 310 performing moving image shooting and a communicable peripheral image processing terminal 320 existing in a predetermined communication network 300. . Communication is in data communication with a peripheral image processing terminal capable of transmitting and receiving data via a wireless telephone network, WLAN, or other wireless network. The image processing terminals may be paired in advance, and transmission and reception of data may be limited only to the paired devices. Here, the image processing terminal 320 is not limited to a digital camera and a digital video camera, and may be a terminal such as a smartphone capable of data communication and having a photographing function. As information to be communicated, release information of peripheral image processing terminals is acquired in real time. Here, the reason for acquiring the release information is that it is considered that the highlight scene is the case where the frequency of shooting still images is high in the event at the same venue. For example, in an event such as an athletic meet or a soccer tournament, the vicinity of the goal scene is particularly exciting, and the shutter is frequently released frequently, and the scene before and after that is appropriate as a highlight scene.

  Subsequently, the imaging unit 110 transmits the obtained digital image signal to the RAM 102 (S205), and temporarily stores the digital image signal. Next, the image processing unit 111 performs the above-described various image quality adjustment processing (S206) on the digital image signal stored in the RAM 102 in accordance with the set photographing setting, and stores the processed digital image signal in the RAM 102 again. . Further, the display control unit 141 reads out the processed digital image signal stored in the RAM 102 or the unprocessed digital image signal and causes the display unit 140 to display the read out digital image signal.

  Further, the encoding processing unit 130 performs encoding processing for encoding the digital image signal processed by the image processing unit 111 (S207).

  Also, the audio processing unit 121 applies various adjustment processes to the digital audio signal input from the microphone unit, and stores the processed digital audio signal in the RAM 102 again (S208).

  Furthermore, as necessary, the encoding process is executed according to a preset audio encoding scheme, and the obtained audio data is stored in the RAM 102. In the following description, although the description of audio data is omitted, it is assumed to be processed together with moving image data.

  Next, the recording / reproducing unit 150 records the encoded moving image data temporarily stored in the RAM 102 as a moving image file compatible with the file system of the recording medium 151 (S209). Further, necessary management information generated by the CPU 101 is included in the moving image file and recorded.

  The CPU 101 causes each block of the imaging apparatus 100 to execute the series of processes until the moving image recording end instruction is input (S210).

  Next, when an instruction to end moving image recording is input from the operation unit 104, the encoding processing unit 130 operates until encoding of the digital image signal until the instruction to end recording is input, and then ends the operation. .

  Further, the recording / reproducing unit 150 also operates until the recording of the encoded moving image data temporarily stored in the RAM 102 is completed to the recording medium 151, and then the operation is ended. The communication unit 160 also ends communication with the peripheral image terminal. The other imaging units 110, the image processing unit 111, the display unit 140, and the display control unit 141 continue the operation.

  Thereafter, highlight extraction processing is performed based on the release information of the peripheral image processing terminal stored in the RAM 102 (S211).

  Here, extraction of a highlight scene will be described with reference to FIG.

  The release information subjected to communication processing with the peripheral image processing terminal in S204 is accumulated, and a histogram in the time axis direction is created. A section of R_Th1 or more is extracted as a highlight scene with respect to the histogram of FIG. The threshold R_Th1 is made variable according to the number of peripheral image processing terminals communicated. That is, by increasing the threshold when the number of terminals communicated is large and decreasing the threshold when small, it is possible to perform control according to the number of image processing terminals. In the example of FIG. 4A, highlight scene 1 (H1_S to H1_E), highlight scene 2 (H2_S to H2_E), and highlight scene 3 (H3_S to H3_E) are extracted with respect to the captured moving image recording section. Do. The above highlight section information is attached to moving image data.

  In the above example, a histogram of release information is generated, and an area exceeding the threshold is determined as a highlight scene, but a highlight scene may be extracted according to the number of releases per unit time. In the example of FIG. 4B, the S4 section and the S7 section exceeding the release number threshold R_Th2 are determined as the highlight scene.

  As described above, the imaging apparatus 100 of the present embodiment records a moving image file on the recording medium 151 in the moving image recording mode.

  By adding highlight information to the moving image data recorded as described above, it is possible to simplify the case of digest reproduction and editing after shooting.

Second Embodiment
In the first embodiment, release information of the peripheral image processing terminal is received in real time at the time of moving image recording, and a highlight scene is detected based on the information and attached to moving image data. The second embodiment is different from the first embodiment in that the image server is accessed and highlight information is added via the server.

  The second embodiment will be described using the flowchart of FIG.

  After capturing a series of moving image data, connection is made to the image server via the network (S501). The network used here may be connected via a wireless telephone network, WLAN, or another wireless network, or a home PC.

  Next, still image data to be used for highlight detection is extracted based on time information and position information (S502). Here, still image data in which captured moving image data, position information, and time information match with each other is extracted from still image data in the image server. That is, image data groups captured at the same event and at the same time are extracted. The number of still image data extracted here corresponds to the release number of the first embodiment.

  Next, a highlight portion is detected by the same method as that of the first embodiment shown in FIG. 4 (S 503), and the detected highlight portion is added to moving image data (S 504), thereby the first embodiment and The same effect is obtained.

(Other embodiments)
The embodiments described above can also be realized as software by a computer (or a CPU, an MPU or the like) of a system or apparatus. Therefore, the computer program itself supplied to the computer in order to realize the above-described embodiment by the computer also implements the present invention. That is, a computer program for realizing the functions of the above-described embodiments is also one of the present invention.

  A computer program for realizing the above-described embodiment may be in any form as long as it is computer readable. For example, it can be configured by an object code, a program executed by an interpreter, script data supplied to an OS, and the like, but is not limited thereto. A computer program for implementing the embodiments described above is provided to a computer by a storage medium or wired / wireless communication. Examples of storage media for supplying the program include magnetic storage media such as flexible disks, hard disks, and magnetic tapes, light / magneto-optical storage media such as MOs, CDs, and DVDs, and nonvolatile semiconductor memories.

  As a method of supplying computer programs using wired / wireless communication, there is a method of using a server on a computer network. In this case, data files (program files) that can be computer programs forming the present invention are stored in the server. The program file may be an executable or source code. Then, the program is supplied by downloading the program file to the client computer that has accessed this server. In this case, it is also possible to divide the program file into a plurality of segment files and distribute the segment files to different servers. That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one of the present invention.

  In addition, a storage medium in which a computer program for realizing the above-described embodiment is encrypted and stored is distributed, key information for decrypting encryption is supplied to a user who satisfies predetermined conditions, and the user's computer is provided. You may allow installation. The key information can be supplied, for example, by downloading it from a home page via the Internet. In addition, a computer program for realizing the above-described embodiment may utilize the function of the OS which is already operated on the computer. Furthermore, a computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to a part of the computer program, or may be executed by a CPU provided in the expansion board or the like. Good.

100 imaging apparatus, 101 CPU, 102 RAM, 103, ROM,
104 operation unit, 110 imaging unit, 111 image processing unit, 120 microphone unit,
121 audio processing unit, 122 speaker unit

Claims (11)

An image processing apparatus for imaging a subject and processing an obtained image signal, comprising:
A moving image processing unit that generates and records a series of moving images;
An information acquisition unit that acquires release information from an image processing terminal other than the image processing apparatus, and the release information is accumulated, and a highlight scene of a moving image generated by the moving image processing unit based on the accumulated information And a highlight extraction unit for extracting the image. The image processing apparatus according to claim 1, wherein the information acquisition unit acquires information from a peripheral image processing terminal capable of transmitting and receiving data to and from the image processing apparatus. The image processing apparatus according to claim 1, wherein the information acquisition unit is connected to an image server and acquires information from image data on the connected server. The image processing apparatus according to claim 1, wherein the information acquisition unit acquires position information from image data on a connected image server. The image processing apparatus according to claim 1, wherein the information acquisition unit acquires imaging time information from image data on a connected image server. The image processing apparatus according to claim 1, wherein the highlight extraction unit generates a histogram based on release information obtained from the information acquisition unit. 7. The image processing apparatus according to claim 6, wherein a highlight scene is determined when the threshold value is exceeded in the histogram. The image processing apparatus according to claim 1, wherein the highlight extraction unit accumulates the number of releases per unit time obtained by the information acquisition unit. The image processing apparatus according to claim 1, wherein when the number of releases per unit time exceeds a threshold, the image processing apparatus is determined as a highlight scene. The image processing apparatus according to claim 9, wherein the threshold value is variable according to the number of image processing terminals communicated. The image processing apparatus according to claim 1, wherein the highlight information extracted by the highlight extraction unit is added to moving image data generated by the moving image processing unit.