WO2014155963A1 - 画像生成装置、撮影装置、画像生成方法及びプログラム - Google Patents
画像生成装置、撮影装置、画像生成方法及びプログラム Download PDFInfo
- Publication number
- WO2014155963A1 WO2014155963A1 PCT/JP2014/001006 JP2014001006W WO2014155963A1 WO 2014155963 A1 WO2014155963 A1 WO 2014155963A1 JP 2014001006 W JP2014001006 W JP 2014001006W WO 2014155963 A1 WO2014155963 A1 WO 2014155963A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- frame
- unit
- information
- person
- Prior art date
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 title claims description 26
- 239000000203 mixture Substances 0.000 claims abstract description 64
- 238000000605 extraction Methods 0.000 claims abstract description 32
- 239000000284 extract Substances 0.000 claims abstract description 15
- 238000007493 shaping process Methods 0.000 claims abstract description 15
- 230000002349 favourable effect Effects 0.000 abstract 1
- 238000001514 detection method Methods 0.000 description 90
- 238000010586 diagram Methods 0.000 description 28
- 230000005236 sound signal Effects 0.000 description 25
- 230000006835 compression Effects 0.000 description 22
- 238000007906 compression Methods 0.000 description 22
- 230000036544 posture Effects 0.000 description 21
- 238000006243 chemical reaction Methods 0.000 description 14
- 239000003086 colorant Substances 0.000 description 9
- 238000003708 edge detection Methods 0.000 description 8
- 238000010191 image analysis Methods 0.000 description 8
- 230000001133 acceleration Effects 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 6
- 230000035945 sensitivity Effects 0.000 description 3
- 230000006837 decompression Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000001454 recorded image Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/387—Composing, repositioning or otherwise geometrically modifying originals
- H04N1/3872—Repositioning or masking
- H04N1/3873—Repositioning or masking defined only by a limited number of coordinate points or parameters, e.g. corners, centre; for trimming
- H04N1/3875—Repositioning or masking defined only by a limited number of coordinate points or parameters, e.g. corners, centre; for trimming combined with enlarging or reducing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/61—Control of cameras or camera modules based on recognised objects
- H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/765—Interface circuits between an apparatus for recording and another apparatus
- H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
- H04N5/772—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera the recording apparatus and the television camera being placed in the same enclosure
Definitions
- the present disclosure relates to an image generation apparatus that generates a new image by cutting out a part of an image taken by a video camera or the like, a photographing apparatus including the image generation apparatus, and a part of the image from the photographed image.
- the present invention relates to a method and a program for cutting out and generating a new image.
- Patent Document 1 describes an imaging apparatus that can acquire a moving image of a subject of interest without focusing on the shooting operation during moving image shooting. Further, Patent Document 1 discloses a configuration in which a specific subject is detected from a captured image, a part of the region including the specific subject is extracted as a cutout image, and the resolution of the cutout image is improved.
- An image generation device of the present disclosure and a photographing device of the present disclosure include a camera work information extraction unit that extracts photographing information of a photographed image, and a new image based on a constraint condition from the photographing information and the photographed image.
- a composition information calculation unit that determines a cutout frame for cutting out, and a composition information shaping unit that uses the shooting information and the cutout frame as attribute information of the shot image.
- an image newly generated from the captured image becomes a preferable image that is easy to view for the viewer, even if there is no skill in image capturing at the time of image capturing. Recorded images and newly generated images can be efficiently recorded and managed.
- FIG. 1 is a perspective view of the video camera according to Embodiment 1.
- FIG. FIG. 2 is a block diagram illustrating a schematic configuration of the video camera according to the first embodiment.
- FIG. 3 is a block diagram showing details of the composition information calculation unit in FIG.
- FIG. 4 is a flowchart for explaining a procedure for generating a new image by cutting out a part of the image from the captured image according to the first embodiment.
- FIG. 5 is a diagram illustrating an example of an image captured by the video camera 100.
- FIG. 6 is a diagram illustrating a result of detecting a person and a motion between frames in the example of the captured image of FIG. FIG.
- FIG. 7 is a diagram illustrating the determination of the cutout frame from the result of detecting the person detection and the motion between frames in the example of the photographed image of FIG.
- FIG. 8 is a diagram illustrating an example of the constraint condition according to the first embodiment.
- FIG. 9 is a diagram for explaining the clipping frame determined in FIG.
- FIG. 10 is a diagram in which the cutout frame of each frame in FIG. 9 is enlarged to the size of the original captured image.
- FIG. 11 is a block diagram illustrating a schematic configuration of the video camera according to the second embodiment.
- FIG. 12 is a block diagram showing details of the composition information calculation unit in FIG. FIG.
- FIG. 13 is a flowchart for explaining a procedure for generating a new image by cutting out a part of an image from a captured image according to the second embodiment.
- FIG. 14 is a diagram for explaining clipping of an image shot by the video camera 100 according to the first embodiment.
- FIG. 15 is a diagram for explaining clipping of an image shot by the video camera 1100 according to the second embodiment.
- FIG. 16 is a block diagram illustrating a schematic configuration of the video camera according to the third embodiment.
- FIG. 17 is a flowchart for explaining a procedure for generating a new image by cutting out a part of an image from a captured image according to the third embodiment.
- FIG. 18 is a diagram for explaining recording of attribute information of a captured image and a newly generated image according to the third embodiment.
- Embodiment 1 As in Patent Document 1, when an area including a specific subject is extracted from a captured image, an image is generated based on the extracted image, and a moving image is generated by connecting them, an image that is easy for the viewer to generate is generated. There are cases where it is not possible.
- the composition may be lost and the image may be difficult for the viewer to see.
- the generated moving image includes a moving image or a flickering moving image.
- the present disclosure has been made in view of the above point, and the purpose of the present disclosure is to generate a new image by cutting out a part of a captured image, and the generated image is transmitted to the viewer.
- An object of the present invention is to provide an image generation apparatus that provides an easy-to-view image, a photographing apparatus including the image generation apparatus, and a method and program for cutting out a part of the image from the photographed image and generating a new image.
- FIG. 1 is a perspective view of a video camera 100 as an example of a photographing apparatus.
- the video camera 100 captures an image, records it, and plays it back. Further, the video camera 100 generates a new image from the captured image.
- FIG. 2 is a block diagram showing a schematic configuration of the video camera 100.
- the video camera 100 includes a lens group 200, an image sensor 201, an image AD converter (Analog Digital Converter) 202 and an image signal processor 203, an image signal compressor 204, a lens control module 205, an attitude detector 206, and an external input unit 207. , Microphone 208, audio AD conversion unit (Analog Digital Converter) 209, audio signal compression unit 210, image signal expansion unit 211, video display unit 212, audio signal expansion unit 213, audio output unit 214, output I / F (Interface) 215, a control unit 300, a clock 301, and a storage unit 302.
- This video camera 100 constitutes an image generation device and a photographing device.
- the lens group 200 is composed of a plurality of lenses and forms an image of incident light from the subject on the image sensor 201.
- the lens group 200 is configured to be able to adjust the distance between the plurality of lenses, and changes the focal length and the zoom magnification by changing the distance between the lenses. Adjustment of the focal length and zoom magnification may be performed manually by the user or by the control unit 300.
- the image sensor 201 converts incident light into an electrical signal.
- the image sensor 201 is composed of a CCD or a CMOS.
- the image sensor 201 outputs an electric signal in an arbitrary range on the imaging surface.
- the image sensor 201 also includes chromaticity space information of the three primary color points, white coordinates, and gain information of at least two of the three primary colors, color temperature information, ⁇ uv (delta uv), and 3 Information such as gamma information of primary colors or luminance signals can also be output. Therefore, the output from the image sensor 201 is input not only to the image AD conversion unit 202 but also to a camera work information extraction unit 315 described later of the control unit 300.
- the image AD converter 202 converts an analog electrical signal output from the image sensor 201 into a digital signal.
- the image signal processing unit 203 amplifies the digital signal output from the image AD conversion unit 202 and performs predetermined processing such as white balance adjustment and color correction.
- the image signal processing unit 203 is configured by an IC for image signal conversion, for example.
- the output from the image signal processing unit 203 is input not only to the image signal compression unit 204 but also to an image analysis unit 313 described later of the control unit 300.
- the image signal compression unit 204 performs predetermined coding conversion on the digital video signal output from the image signal processing unit 203 to compress the data amount.
- Specific examples of encoding conversion include encoding schemes such as MPEG2, MPEG4, H264 / MPEG4 and AVC.
- An output from the image signal compression unit 204 is input to the control unit 300.
- the image signal compression unit 204 is configured by an IC for image signal compression / decompression, for example.
- the image signal compression unit 204 receives not only a digital video signal output from the image signal processing unit 203 but also a signal output from a composition information shaping unit 317 (to be described later) of the control unit 300, and a predetermined signal is input to the input signal.
- the data conversion is performed to compress the data amount.
- the lens control module 205 detects the state of the lens group 200 and operates the lens group 200.
- the lens control module 205 includes a lens control motor and a lens position sensor.
- the lens position sensor detects a distance or a positional relationship between a plurality of lenses constituting the lens group 200.
- the lens position sensor outputs the detection signal to the control unit 300.
- the lens control module 205 has two types of lens control motors. One lens control motor moves the lens group 200 in the optical axis direction based on a control signal from the control unit 300. Thereby, the distance between the plurality of lenses of the lens group 200 is changed, and the focal length and zoom magnification of the lens group 200 are adjusted.
- the other lens control motor moves at least one lens of the lens group 200, for example, an image blur correction lens, in a plane orthogonal to the optical axis, based on a control signal from the control unit 300. Thereby, image blur is corrected.
- the posture detection unit 206 detects the posture state of the video camera 100 main body.
- the posture detection unit 206 includes an acceleration sensor, an angular velocity sensor, and an elevation angle / decline angle sensor. With these sensors, it is possible to recognize the posture of the video camera 100 at the time of shooting.
- the acceleration sensor and the angular velocity sensor can detect the postures in the three orthogonal directions, that is, the posture of the video camera 100 in the vertical direction, the front-rear direction, and the horizontal direction.
- position detection part 206 is not restricted to said structure, You may be comprised only with either of the said sensors, and may have sensors other than the said sensor.
- the external input unit 207 inputs information from the outside to the video camera 100.
- a signal from the external input unit 207 is input to the control unit 300.
- the signal from the external input unit 207 is input only to the camera work information extraction unit 315 of the control unit 300, but is also input to a part corresponding to the input operation of the lens control unit 311 and the like.
- various information from the outside is input to the video camera 100 via the external input unit 207.
- the external input unit 207 includes an input button that is one of input interfaces for receiving input of information from the user, and a tripod sensor that detects whether or not the video camera 100 is installed on a tripod.
- the tripod sensor is composed of a switch provided at a portion where the tripod of the video camera 100 is fixed. With a tripod sensor, it can be determined whether or not shooting is performed using a tripod.
- the device is not limited to an input button or a tripod sensor, and any device such as a touch panel, a joystick, or the like that can accept instructions from the user may be used.
- the microphone 208 collects sound around the video camera 100 when the video camera 100 is taking an image, converts it into an electrical signal, and outputs it as an analog audio signal.
- the audio AD conversion unit 209 converts the analog audio signal data output from the microphone 208 into audio data that is a digital audio signal.
- the audio signal compression unit 210 encodes the digital audio signal output from the audio AD conversion unit 209 using a predetermined encoding method. For encoding, there are methods such as MP3 (MPEG Audio Layer-3) and AAC (Advanced Audio Coding).
- the audio signal compression unit 210 is composed of an IC for audio signal compression, for example.
- the image signal decompression unit 211 decodes the video signal output from the control unit 300.
- the output from the image signal expansion unit 211 is input to the video display unit 212.
- the video display unit 212 newly generates an image recorded in the video camera 100, an image captured in real time by the video camera 100, an image captured in the past, or a part of an image captured from an image captured later. Display the selected image.
- the video display unit 212 displays various types of information such as information related to shooting and device information.
- the video display unit 212 is configured by, for example, a touch panel type liquid crystal display.
- the touch panel video display unit 212 also functions as the external input unit 207.
- the image signal expansion unit 211 is constituted by an IC for image signal expansion, for example.
- the audio signal decompressing unit 213 decodes the audio signal output from the control unit 300.
- the audio signal expansion unit 213 is configured by an audio signal expansion IC, for example.
- the output from the audio signal expansion unit 213 is input to the audio output unit 214.
- the audio output unit 214 outputs audio accompanying the image.
- the audio output unit 214 outputs a warning sound to notify the user from the video camera 100.
- the audio output unit 214 is configured by a speaker, for example.
- the output I / F 215 is an interface for outputting a video signal from the video camera 100 to the outside.
- the output I / F 215 is a cable interface when the video camera 100 and an external device are connected by a cable, a memory card interface when a video signal is recorded on a memory card, or the like.
- the control unit 300 controls the entire video camera 100.
- the control unit 300 includes an image sensor 201, an image signal processing unit 203, an image signal compression unit 204, a lens control module 205, an attitude detection unit 206, an external input unit 207, an audio AD conversion unit 209, an audio signal compression unit 210, an image signal. Signals can be exchanged with the expansion unit 211, the audio signal expansion unit 213, the output I / F 215, the storage unit 302, and the like.
- control unit 300 is configured by a CPU.
- the control unit 300 executes various controls of the video camera 100 by reading and executing a program stored in the storage unit 302.
- control executed by the control unit 300 include, for example, control of the focal length and zoom of the lens group 200, processing of input signals from the posture detection unit 206 and the external input unit 207, image signal processing unit 203, and image This is operation control of ICs such as the signal compression unit 204, the audio signal compression unit 210, the image signal expansion unit 211, and the audio signal expansion unit 213.
- the signal is appropriately AD-converted or DA-converted between the control unit 300 and the lens control module 205 or the like.
- the control unit 300 can also be configured by an integrated circuit such as an IC.
- the clock 301 supplies a clock signal serving as a reference for processing operation to the control unit 300 or the like operating in the video camera 100.
- a single clock or a plurality of clocks can be used as the clock 301 depending on an integrated circuit to be used and data to be handled. Further, an arbitrary multiple of the clock signal of one oscillator may be used.
- the storage unit 302 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive).
- the ROM stores a program processed by the control unit 300 and various data for operating the program.
- the RAM is used as a memory area used when executing a program processed by the control unit 300.
- the RAM can also be used as a memory area of the IC.
- the HDD includes video data encoded and converted by the image signal compressing unit 204, audio data encoded and converted by the audio signal compressing unit 210, an image generated by extracting a part of an image from a captured image, which will be described later, and the like. Save various data. Further, the HDD stores a program executed by the control unit 300.
- the storage unit 302 is not limited to the HDD, and may be stored in a semiconductor memory, or may include a portable recording medium such as an SD memory card, a CD-ROM, or a DVD.
- control unit 300 will be described in detail.
- a configuration necessary for generating a new image by cutting out a part of the image from the captured image in the control unit 300 will be described.
- the control unit 300 has a configuration for exhibiting the same function as that of a general video camera, for example, a configuration for recording or reproducing a captured image.
- the control unit 300 includes a lens control unit 311 that controls the lens control module 205, an imaging control unit 312 that controls the imaging element 201, an image analysis unit 313 that analyzes an output from the image signal processing unit 203, and audio AD conversion.
- a sound analysis unit 314 that analyzes the output from the unit 209, a camera work information extraction unit 315 that generates a shooting state of the video camera 100, and an image that is shot by the video camera 100 is analyzed and analyzed, and a newly generated image
- a composition information calculation unit 316 that determines a clipping frame, a composition information shaping unit 317 that newly generates an image using the determined clipping frame, a multiplexing unit 318 that multiplexes a video signal and an audio signal, and playback A playback unit 319 is provided.
- the control unit 300 reads and executes a program stored in the storage unit 302, thereby realizing various processes described later.
- the lens control unit 311 receives the detection signal of the lens position sensor of the lens control module 205 and the detection signal of various sensors of the posture detection unit 206.
- the lens control unit 311 outputs a control signal for properly arranging the lens group 200 to the lens control motor based on these detection signals and information from other components such as the image sensor 201.
- the lens control unit 311 performs zoom control, focus control, image blur correction control, and the like.
- the lens control unit 311 outputs a control signal for the lens group 200 to the camera work information extraction unit 315.
- detection signals from various sensors of the posture detection unit 206 are also output to the camera work information extraction unit 315.
- the imaging control unit 312 controls the operation of the imaging element 201.
- the exposure amount at the time of shooting, the shooting speed, sensitivity, and the like are controlled.
- the control signal output from the imaging control unit 312 is output not only to the imaging element 201 but also to the camera work information extraction unit 315.
- the image analysis unit 313 extracts image features based on the video data from the image signal processing unit 203.
- the image analysis unit 313 detects image color information, for example, information on the distribution of colors included in the image and white balance information. The color distribution can be detected by confirming the color information included in the data forming the digital video signal. Further, the image analysis unit 313 performs face detection from an image when a human face is included in the image. Face detection can be realized by using pattern matching or the like.
- the voice analysis unit 314 analyzes the voice data from the voice AD conversion unit 209 and extracts characteristic sounds.
- the characteristic sounds here include, for example, a photographer's voice, pronunciation of a specific word, cheers, gunshots, and the like. These sounds can be extracted by a method of registering in advance a unique frequency of these sounds and discriminating them by comparison with them. In addition to this, for example, it may be determined that the sound is characteristic when the sound input level is equal to or higher than a predetermined level.
- the camera work information extraction unit 315 extracts shooting information regarding the image being shot.
- the camera work information extraction unit 315 receives output signals from the image sensor 201, posture detection unit 206, external input unit 207, lens control unit 311, image capture control unit 312, image analysis unit 313, and audio analysis unit 314. Yes.
- the camera work information extraction unit 315 extracts shooting information based on these output signals.
- the shooting information related to this image includes information related to shooting such as the state of the shooting device at the time of shooting the image and camera work, information related to the subject and background included in the image itself, and information related to the sound attached to the image.
- information regarding the photographing apparatus at the time of photographing include focal length, zoom magnification, exposure, photographing speed, sensitivity, color space information of three primary colors, white balance, gain information of at least two of the three primary colors, color temperature Information, ⁇ uv (delta uv), gamma information of three primary colors or luminance signals, color distribution, face recognition information, camera posture, for example, acceleration, angular velocity, elevation angle, depression angle, and the like, shooting time, for example, shooting start time, end Time, and includes shooting index information, user input content, frame rate, sampling frequency, and the like.
- the camera work information extraction unit 315 extracts a focal length and a zoom magnification as shooting information based on a control signal from the lens control unit 311. Further, the camera work information extraction unit 315 detects the camera posture such as acceleration, angular velocity, elevation angle, depression angle, and the like based on the detection signal of the posture detection unit 206, and swings the camera direction left and right from the detected camera posture. The operation of the camera of the video camera 100 at the time of shooting such as a certain pan or tilt tilting the camera up and down is extracted as shooting information. Furthermore, a portion that is shot while the video camera 100 is operated after the video camera 100 is operated can be extracted as shooting information. As described above, the camera work information extraction unit 315 may extract the shooting information from the input signal itself, or may extract the shooting information by combining or analyzing the input signal. .
- the composition information calculation unit 316 determines a cut-out frame of a newly generated image based on the shooting information extracted by the camera work information extraction unit 315. Details of the determination of the cutout frame will be described later.
- the composition information shaping unit 317 generates a new image from the captured image output from the image signal processing unit 203 based on the clipping frame determined by the composition information calculation unit 316.
- the generated image is output to the image signal compression unit 204. Details of image generation will be described later.
- the multiplexing unit 318 multiplexes the encoded video data from the image signal compression unit 204 and the encoded audio data from the audio signal compression unit 210 and outputs the multiplexed data.
- the data multiplexed by the multiplexing unit 318 is stored in the storage unit 302.
- the multiplexing method is, for example, the MPEG TS (Transport Stream) method. However, it is not limited to this.
- the processes of the camera work information extraction unit 315, the composition information calculation unit 316, the composition information shaping unit 317, and the multiplexing unit 318 are sequentially performed at the time of photographing, specifically during photographing or immediately after photographing.
- the playback unit 319 executes playback of an image cut out based on a user input after the shooting is completed. Specifically, the reproduction unit 319 reads the multiplexed data stored in the storage unit 302, and the encoded video data and the encoded audio data of the part to be reproduced are the image signal expansion unit 211 and the audio signal expansion unit 213, respectively. Output to. The output encoded video data and encoded audio data are decoded by the image signal expansion unit 211 and the audio signal expansion unit 213, respectively, and output from the video display unit 212 and the audio output unit 214. Thus, the reproduction of the image cut out from the image taken from the image is executed.
- FIG. 3 is a diagram showing a detailed configuration of the composition information calculation unit 316.
- the shooting information output from the camera work information extraction unit 315 is used as an input signal.
- a human detection unit 401 that detects which person is included in the captured image from the input shooting information, and when the person is included, detects which face is facing, who is the person, and the like.
- Motion detection unit 402 that detects whether a person or an object is moving between frames of a frame, which is a unit of one frame of an image, and whether a characteristic figure is moving in the preceding and following frames, that is, video
- An edge detection unit 403 that detects whether the camera 100 is fixed and shooting, a cutout frame based on the shooting information, the detection result of the human detection unit 401, the detection result of the motion detection unit 402, and the detection result of the edge detection unit 403
- a cutout frame determination unit 404 for determining
- FIG. 4 is a flowchart for explaining a procedure for cutting out a part of an image from a captured image and generating a new image.
- FIG. 5 is a diagram illustrating an example of an image photographed by the video camera 100.
- the captured image is shown by a frame which is a unit of one frame of the image.
- the images are examples taken in the order of frame F1, frame F2, frame F3, frame F4, frame F5, frame F6, frame F7, frame F8, and frame F9.
- This image was taken with the video camera 100 fixed on a tripod.
- the frame F1 the person A is standing on the slightly left side of the screen on which the person A is photographed, and the whole body is photographed.
- the frame F2 faces the right with respect to the frame F1 at the position where the person A stands in the frame F1 as viewed from the video camera 100 that is shooting.
- the person B can be seen at the right end of the captured screen with respect to the frame F2.
- the person B is walking from the right end of the screen being shot toward the left side with respect to the frame F3.
- the person B walks to the right of the person A with respect to the frame F4.
- the person B stops on the right side of the person A with respect to the frame F5.
- Person A turned his face to the left.
- the frame F7 with respect to the frame F6 the person A turns to the left and starts walking toward the left end.
- the person B faces the front with respect to the frame F7.
- the frame F9 is about to disappear from the frame F8 at the left end of the screen on which the person A is photographed.
- the camera work information extraction unit 315 extracts shooting information about the image being shot.
- the camera work information extraction unit 315 receives output signals from the image sensor 201, posture detection unit 206, external input unit 207, lens control unit 311, image capture control unit 312, image analysis unit 313, and audio analysis unit 314. Yes.
- the camera work information extraction unit 315 extracts shooting information based on these output signals.
- the shooting information related to this image includes information related to shooting such as the state of the shooting device at the time of shooting the image and camera work, information related to the subject and background included in the image itself, and information related to the sound attached to the image.
- information regarding the photographing apparatus at the time of photographing include focal length, zoom magnification, exposure, photographing speed, sensitivity, color space information of three primary colors, white balance, gain information of at least two of the three primary colors, color temperature Information, ⁇ uv (delta uv), gamma information of three primary colors or luminance signals, color distribution, face recognition information, camera posture, for example, acceleration, angular velocity, elevation angle, depression angle, and the like, shooting time, for example, shooting start time, end Time, and includes shooting index information, user input, frame rate, sampling frequency, and the like.
- the camera work information extraction unit 315 extracts a focal length and a zoom magnification as shooting information based on a control signal from the lens control unit 311. Further, the camera work information extraction unit 315 detects the camera posture such as acceleration, angular velocity, elevation angle, depression angle, and the like based on the detection signal of the posture detection unit 206, and swings the camera direction left and right from the detected camera posture. The operation of the camera of the video camera 100 at the time of shooting such as a certain pan or tilt tilting the camera up and down is extracted as shooting information. Furthermore, a portion that is shot while the video camera 100 is operated after the video camera 100 is operated can be extracted as shooting information. As described above, the camera work information extraction unit 315 may extract the shooting information from the input signal itself, or may extract the shooting information by combining or analyzing the input signal. .
- Step S402 Human detection, motion detection, and edge detection are performed using the imaging information extracted by the camera work information extraction unit 315.
- the person detection unit 401 detects a person from the shooting information.
- the person may be detected simply by detecting whether the person is a person, or by registering information on a person to be recognized in advance in the storage unit 302 and comparing it with the registered information. May be detected. Further, the direction of whether the detected person is facing the front or the back is also detected.
- the person detection unit 401 may detect a person shown in the image from the perspective of the captured image. For example, if there are a person A and a person B shown in the image, and the person A is shown larger than the person B, the person A is arranged in the front and the person B is arranged in the rear. May be detected.
- the motion detection unit 402 identifies a portion that moves between frames, and identifies in what direction and how fast the portion moves from the previous frame to the subsequent frame.
- a technique for detecting “motion” can be realized by using a technique such as a motion vector between frames.
- the edge detection unit 403 performs an analysis for identifying a background portion in the captured image. Specifically, in the process of specifying “motion”, a portion where the “motion” is small is recognized as the background. Thereby, the background portion can be identified.
- the edge detection unit 403 may specify the background portion based on the strength of the contour of the subject of the photographed image and the texture. Further, the background portion may be specified by analyzing information about the color of the photographed image, for example, information regarding the color distribution information of each of R, G, and B as a histogram. Further, the edge detection unit 403 detects whether or not a characteristic figure such as a window frame of a building or a road sign is moving in the specified background portion, that is, whether or not the video camera 100 is fixed.
- FIG. 6 is a diagram for explaining a result of detecting a person and a motion between frames in the example of the photographed image of FIG.
- the person A is detected and detected as a person, and the detection result is displayed surrounded by a person detection frame w11.
- the frame F2 it is detected that the person A is a person continuously by human detection, and the detection result is displayed surrounded by a human detection frame w21.
- the person detection detects that the person B is a person in addition to the person A, and the detection result is displayed in the person detection frame w41 for the person A and in the person detection frame w42 for the person B. ing. Further, in the frame F4, it is detected by motion detection that the person B has walked to the left with respect to the frame F3, and the detected “motion” is displayed as a motion vector V4.
- the person A and the person B are continuously detected by the human detection, and the detection results are displayed surrounded by the human detection frame w51 and the human detection frame w52, respectively. Further, in the frame F5, the person B continues to move to the left with respect to the frame F4, so that it is detected by motion detection, and the detected “motion” is displayed by the motion vector V5.
- the person A and the person B are continuously detected by the human detection, and the detection results are displayed surrounded by the human detection frame w61 and the human detection frame w62, respectively.
- the person A and the person B are continuously detected by the person detection, and the detection results are displayed surrounded by the person detection frame w71 and the person detection frame w72, respectively. Furthermore, in the frame F7, since the person A has moved to the left with respect to the frame F6, it is detected by motion detection, and the detected “motion” is displayed as a motion vector V7.
- the person A and the person B are continuously detected by the person detection, and the detection results are displayed surrounded by the person detection frame w81 and the person detection frame w82, respectively.
- frame F9 the person A is not detected because it is going to disappear at the left end of the captured screen, and the person B is continuously detected as a person, and the detection result is surrounded by a person detection frame w92. Is displayed. Further, in the frame F9, the person A moves to the left with respect to the frame F8, so that it is detected by motion detection, and the detected “motion” is displayed by the motion vector V9.
- the human detection frame is detected by the human detection unit 401, and the motion vector is detected by the motion detection unit 402.
- the detection by the edge detection unit 403 was not performed.
- the cutout frame determination unit 404 determines a cutout frame from the photographing information, the detection result of the person detection unit 401, the detection result of the motion detection unit 402, and the detection result of the edge detection unit 403.
- FIG. 7 is a diagram for explaining a method for determining a cutout frame from the result of detecting the human detection result of FIG. 6 and the motion between frames in the example of the photographed image of FIG.
- FIG. 8 is a diagram showing an example of the constraint condition.
- the types of constraint conditions there are a constraint on the frame itself, a constraint for each frame, a constraint extending between frames, and the like.
- the frame itself there is a restriction on the size (size) of the frame of the image to be cut out.
- the frame to be cut out is 1/4 or more of the captured screen.
- the amount of change in the size (size) of the frame of the image to be cut out between frames for example, the amount of change in the size is within W% (W is a decimal number).
- Restrictions for each frame are restrictions related to the composition of the frame.
- Restrictions regarding the frame composition include a three-part composition, a triangular composition, a diagonal composition, a Hinomaru composition, and a symmetry composition.
- the screen is divided into 9 equal parts by dividing the screen vertically and horizontally by 2 horizontal lines and 2 vertical lines with the vertical and horizontal sides of the screen to be cut at equal intervals. This is a constraint for placing an important element in the composition at the intersection of.
- Triangular composition is a restriction in which tall trees and buildings are arranged in the center of the screen and arranged in a triangle.
- Diagonal composition is a constraint in which a linearly extending subject such as a row of trees, a road or a tree branch is placed on the diagonal.
- the Hinomaru composition is a restriction that places the subject in the middle like a Hinomaru.
- the symmetry composition is a constraint that subjects are arranged symmetrically or vertically. The composition is determined so as to be applied to any one of the constraints on the composition of the frame.
- ⁇ ⁇ Restrictions between frames are restrictions related to camera work between frames.
- the restrictions on the camera work between frames are the left and right moving time of the video camera being shot, for example, S1 to S2 seconds, the vertical moving time, for example, S3 to S4 seconds, zoom reduction time, for example,
- There are constraints such as S5 seconds or more and S6 seconds or less, zoom expansion time, for example, S7 seconds or more and S8 seconds or less (S1, S2, S3, S4, S5, S6, S7, S8 are decimal numbers).
- the constraint conditions are stored in the storage unit 302. All the constraints regarding the camera work between frames are determined so as to satisfy the constraints.
- the cutout frame is determined so as to satisfy these constraints. If the composition cannot be determined so that all the constraints of the frame itself, the constraints for each frame, and the constraints between frames are satisfied, give priority to each constraint so that the constraint conditions are satisfied as much as possible. Determine the composition.
- the priority order is a restriction of the frame itself, a restriction between frames, and a restriction for each frame as the order of priority.
- the photographed image is photographed with a tripod fixed as a photographing condition. Therefore, when the image is cut out, it is determined on the assumption that the image is fixed.
- the cutout frame is determined from the person detection frame w11.
- the cutout frame is preferably 1 ⁇ 4 or more of the captured screen, and the person detection frame w11 stands with the person A facing the front.
- the person A is cut out so as to have a composition according to the symmetry composition, for example, the person A is placed at the center above the chest only.
- a frame n1 is determined.
- the restriction on the size of the cutout image frame which is a restriction condition, and the size of the cutout image frame between the cutout frame n1 cut out in the frame F1. Due to restrictions on the amount of change in the amount, restrictions on the composition of the frame, and restrictions on the camera work between the frames, the composition in which the person A is cut from the whole body only above the chest and the person A is cut out and placed on the left side of the frame, for example
- the cutout frame n2 is determined so as to obtain a three-part composition.
- the person A is continuously detected by human detection, and the detected result is displayed surrounded by the person detection frame w31. Therefore, the same cutout frame as the cutout frame n2 is set as the cutout frame n3. decide.
- a cutout frame is determined from the human detection frame w41, the human detection frame w42, and the motion vector V4.
- the restriction on the size of the frame of the image to be cut out, and the amount of change in the size of the frame of the image to be cut out between the cut-out frame n2 cut out in the frame F2 The cut-out frame n4 is determined from the left and right moving times with the above-mentioned restrictions and the restrictions regarding camera work between frames.
- the face of the person B is not cut out so that all the faces of the person B are included in the cut-out frame n2, but the restriction on the interframe camera work is given priority over the restriction on the frame composition.
- a human detection frame w51, a human detection frame w52, and a motion vector V5 are displayed. Restriction on the size of the frame of the image to be cut out, which is a restriction condition, restriction on the amount of change in the size of the frame of the image to be cut out with the cut-out frame n4 cut out on the frame F4, restriction on the composition of the frame, camera work between frames
- the cut-out frame n5 is determined from the restrictions regarding.
- the cutout frame at the same position as the cutout frame n5 is determined as the cutout frame n6.
- a cutout frame is determined from the human detection frame w71, the human detection frame w72, and the motion vector V7. Since both the person A and the person B are cut out in the upper part from the chest and the person A is moving, the restriction on the amount of change in the size of the frame to be cut out between the cut-out frame n6 cut out in the frame F6 is limited.
- the cutout frame n7 is determined with priority.
- the cutout frame at the same position as the cutout frame n7 is determined as the cutout frame n8.
- FIG. 9 is a diagram for explaining the cut-out frame determined in FIG.
- the cutout frame determination unit 404 determines the cutout frame for each frame as described above.
- Step S404 The composition information shaping unit 317 cuts the output signal of the image signal processing unit 203 with the cutout frame cut out by the composition information calculation unit 316, and expands the cut image to the same size as the captured image, thereby creating a new image. Is generated. The newly generated image is output to the image signal compression unit 204.
- FIG. 10 is a diagram in which the cutout frame of each frame in FIG. 9 is enlarged to the size of the original photographed image.
- a frame NF1 a frame NF2, a frame NF3, a frame NF4, a frame NF5, a frame NF6, a frame NF7, a frame NF8, and a frame NF9 are generated.
- the video camera 100 described in this embodiment generates a new image by cutting out part of the captured image.
- the generated new image is cut out from the photographed image based on the constraint condition recorded in the storage unit 302. For this reason, an image newly generated from the photographed image is a more preferable image even if there is no skilled photographing technique or the like when photographing an image with the video camera 100.
- constraint conditions described in the present embodiment are merely examples, and the present invention is not limited to this. At least one of the above-described constraint conditions may be used as a constraint condition, or another constraint condition may be used.
- a constraint condition for example, an S-shaped composition in which a curved line such as a road or a river is taken in and arranged in an S-shape, or a C-shaped composition in which a curved line such as a road or river is taken in There is a figure composition. 5 was only a person, but when extracting from an image in which a plurality of objects such as a person and a car are moving, the determination of the extraction frame is performed from the center of gravity of all detected persons and objects.
- a cutout frame may be determined, or a person or object to be cut out may be given a priority in advance, and the cutout frame may be determined based on the priority. Further, the priority may be specified from the external input unit 207.
- the newly generated image is multiplexed with the image captured by the multiplexing unit 318, stored in the storage unit 302, and then played back by the playback unit 319.
- the data may be reproduced from the reproduction unit 319 before being stored in the unit 302.
- the size of the newly generated image is enlarged to the size of the captured image, but it is not necessary to be the size of the captured image, and the size may be designated from the external input unit 207. You may decide in advance.
- the image when a new image is generated from a captured image, the image may not be smooth between frames of the generated new image even if the constraint condition is satisfied. For example, there are a case where the zoom speed of the video camera 100 that is shooting is faster than a predetermined speed, or a case where the movement of a person or an object is faster than the predetermined speed.
- an interpolation process for smoothing between frames of a new image to be generated will be described.
- FIG. 11 is a block diagram showing a schematic configuration of the video camera 1100 according to the second embodiment.
- the video camera 1100 differs from the video camera 100 of Embodiment 1 in that a cutout frame for generating a new image is determined.
- the basic configuration of the video camera 1100 is substantially the same as that of the video camera 100. Therefore, the same configurations as those of the first embodiment are denoted by the same reference numerals, description thereof is omitted, and different portions are mainly described.
- a video camera 1100 differs from the video camera 100 of the first embodiment in the configuration of the control unit 1300.
- the control unit 1300 differs from the control unit 300 of the first embodiment in the configuration of the composition information calculation unit 1316.
- FIG. 12 is a block diagram showing details of the configuration of the composition information calculation unit 1316.
- the composition information calculation unit 1316 performs interpolation processing on the cutout frame so that the cutout frame can be smoothly reproduced after the cutout frame is temporarily determined by the cutout frame determination unit 404 and then the cutout image can be smoothly reproduced.
- the cut-out frame interpolation processing uses spline interpolation processing or Lagrangian interpolation processing.
- FIG. 13 is a flowchart for explaining a procedure for cutting out a part of an image from a captured image and generating a new image.
- step S1301 is added between steps S403 and S404.
- step S1301 after the cutout frame determining unit 404 temporarily determines the cutout frame, the cutout frame is interpolated so that the cutout image can be smoothly reproduced from the continuity of the cutout frame between frames. Spline interpolation and Lagrange interpolation are used for interpolation of the cutout frame.
- FIG. 14 is a diagram for explaining clipping of an image shot by the video camera 100. That is, this is an example of a case where the clipping frame is not interpolated.
- FIG. 15 is a diagram for explaining clipping of an image shot by the video camera 1100. That is, this is an example when the cutout frame is interpolated.
- the upper part of FIG. 14 is an image taken by the video camera 100, and the image is an example taken in the order of the frame F21, the frame F22, and the frame F23.
- the upper part of FIG. 15 is an image photographed by the video camera 1100, and is an image photographed in the same manner as the video camera 100.
- the cutout frame is determined as described in the first embodiment.
- the cutout frame w21 is determined for the frame F21
- the cutout frame w22 is determined for the frame F22
- the cutout frame w23 is determined for the frame F23.
- the middle part of FIG. 14 shows a cutout frame extracted for each frame.
- the cutout frame of each frame is expanded to the size at the time of shooting, and new images NF21, NF22, and NF23 are generated.
- the size of the person changes abruptly between frames.
- the size of a person between frames is gradually changed.
- the cutout frame is determined as described in the first embodiment. At this time, the determined cutout frame is not fixed but is temporarily determined.
- the cutout frame w21 is determined for the frame F21
- the cutout frame w22 is determined for the frame F22
- the cutout frame w23 is determined for the frame F23. Using the cutout frame w22 of the frame F22 as a reference, the cutout frame w21 of the frame F21 and the cutout frame w23 of the frame F23 are interpolated.
- the cutout frame w21 of the frame F21 and the cutout frame w23 of the frame F23 are interpolated.
- the cutout frame w21 of the frame F21 is interpolated in the interpolation direction i1 to obtain a cutout frame w31.
- the cutout frame w23 of the frame F23 is interpolated in the interpolation direction i3 to obtain a cutout frame w33.
- the middle part of FIG. 15 shows a cutout frame extracted for each frame.
- the cutout frame of each frame is enlarged to the size at the time of shooting, and frames NF31, NF22, and NF33, which are new images, are generated. Compared with the new image in the lower part of FIG. 14, the size of the person changes gradually between frames, that is, the image is smooth.
- the video camera 1100 described in this embodiment generates a new image by cutting out a part of the captured image.
- the generated new image is cut out from the captured image based on the constraint condition recorded in the storage unit 302, and further subjected to interpolation processing between frames. Therefore, an image newly generated from the photographed image becomes a smoother and more natural image even without a skillful photographing technique or the like when photographing an image with the video camera 1100.
- FIG. 16 is a block diagram illustrating a schematic configuration of the video camera 2100 according to the third embodiment.
- the video camera 2100 is different from the video camera 100 of Embodiment 1 in that attribute information for generating a new image is recorded without generating a new image.
- the basic configuration of the video camera 2100 is basically the same as that of the video camera 100. Therefore, the same configurations as those of the first embodiment are denoted by the same reference numerals, description thereof is omitted, and different portions are mainly described.
- the video camera 2100 is different from the video camera 100 in Embodiment 1 in the configuration of an image signal processing unit 2203, an image signal compression unit 2204, a control unit 2300, and a storage unit 2302.
- the image signal processing unit 2203 is different in that the output digital video signal is not input to the composition information shaping unit 2317 of the control unit 2300. This is because the control unit 2300 does not generate a new image.
- the image signal compression unit 2204 is different in that the digital video signal output from the control unit 2300 is not input. This is because the control unit 2300 does not generate a new image.
- the control unit 2300 differs from the control unit 300 of the first embodiment in the configuration of a camera work information extraction unit 2315, a composition information shaping unit 2317, a multiplexing unit 2318, and a reproduction unit 2319.
- the camera work information extraction unit 2315 is different in that the extracted shooting information is input to the composition information shaping unit 2317.
- the composition information shaping unit 2317 outputs the shooting information extracted by the camera work information extraction unit 2315 and the clipping frame determined by the composition information calculation unit 316 to the multiplexing unit 2318 as attribute information.
- the multiplexing unit 2318 records the attribute information together with the captured image in the storage unit 2302.
- the playback unit 2319 executes playback of a new image based on the captured image recorded in the storage unit 2302 and the attached information.
- FIG. 17 is a flowchart for explaining a procedure for generating a new image by cutting out a part of an image from a captured image according to the third embodiment.
- step S404 is changed to step S1701 after the cutout frame is determined in step S403.
- the composition information shaping unit 2317 outputs the shooting information extracted by the camera work information extraction unit 2315 and the clipping frame determined by the composition information calculation unit 316 to the multiplexing unit 2318 as attribute information.
- the multiplexing unit 2318 multiplexes the photographed image, sound accompanying the image, and attribute information into one data.
- the multiplexed data is recorded in the storage unit 2302.
- FIG. 18 is a diagram for explaining recording of attribute information of a captured image and a newly generated image.
- attribute information is recorded as information attached to the photographed image data.
- shooting information and the determined clipping frame are recorded.
- the determined clipping frame is based on the upper left corner of the original captured image, and the upper left corner of the rectangular clipping frame is the information on the offset amount in the horizontal and vertical directions, and the horizontal and vertical directions of the clipping frame. Size information.
- the attribute information, the photographed image, and the sound accompanying the image can be stored in association with each other in the storage unit 2302, they may be stored as individual data.
- the association between the attribute information and the captured image and the sound accompanying the image may be, for example, newly generating another data indicating the relationship between the attribute information and the captured image or image.
- Corresponding audio may be stored as one group in the file system management when recording in the storage unit 2302, for example, by storing it in the same directory in the file system.
- attribute information may be provided only in a specific frame of the captured image, but the present embodiment is not limited to this.
- attribute information may be provided for all frames.
- a reference frame at the time of encoding may be selected, and attribute information may be provided only for that frame.
- the captured image, the sound accompanying the image, and the attribute information may be multiplexed and then encoded.
- the encoded captured image and sound associated with the image and attribute information may be multiplexed.
- the attribute information may or may not be encoded.
- the video camera 2100 described in this embodiment records the shooting information of the shot image and the cutout frame for cutting out a part of the shot image as the attached information of the shot image.
- a new image is reproduced based on the recorded image and the attached information.
- the photographed image and the newly generated image can be efficiently recorded and managed.
- This disclosure can be applied to a video camera that captures and records an image.
- DESCRIPTION OF SYMBOLS 100, 1100, 2100 Video camera 200 Lens group 201 Image pick-up element 202 Image AD conversion part 203, 2203 Image signal processing part 204, 2204 Image signal compression part 205 Lens control module 206 Posture detection part 207 External input part 208 Microphone 209 Voice AD conversion Unit 210 audio signal compression unit 211 image signal expansion unit 212 video display unit 213 audio signal expansion unit 214 audio output unit 215 output I / F 300, 1300, 2300 Control unit 301 Clock 302, 2302 Storage unit 311 Lens control unit 312 Imaging control unit 313 Image analysis unit 314 Audio analysis unit 315, 2315 Camera work information extraction unit 316, 1316 Composition information calculation unit 317, 2317 Composition information Shaping unit 318, 2318 Multiplexing unit 319, 2319 Playback unit
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Studio Devices (AREA)
- Television Signal Processing For Recording (AREA)
- Editing Of Facsimile Originals (AREA)
Abstract
Description
特許文献1のように、撮影された画像から特定の被写体を含む領域を抽出し、抽出した画像に基づいて画像を生成し、それらをつなぎ合わせて動画を生成すると、視聴者に見やすい画像が生成できない場合がある。
図1は、撮影装置の一例としてのビデオカメラ100の斜視図である。ビデオカメラ100は画像を撮影し、それを記録し、再生するものである。また、ビデオカメラ100は撮影した画像から新たな画像を生成する。
次に、撮影された画像から画像の一部を切り出して新たな画像を生成する手順を、撮影された画像の一例を用いて説明する。撮影された画像から画像の一部を切り出して、新たな画像を生成する処理は、カメラワーク情報抽出部315と構図情報計算部316と構図情報整形部317で行う。
以上により、本実施の形態で説明するビデオカメラ100は、撮影した画像の一部を切り出して新たな画像を生成する。その生成される新たな画像は、記憶部302に記録された制約条件に基づいて、撮影された画像から切り出される。そのため、ビデオカメラ100での画像撮影時に熟練した撮影技術等がなくても、撮影した画像から新たに生成される画像は、より好ましい画像となる。
実施の形態1において、撮影した画像から新たな画像を生成する際に、制約条件を満たしたとしても、生成された新たな画像のフレーム間で画像が滑らかにならない場合がある。例えば、撮影しているビデオカメラ100のズーム速度が所定速度よりも速い場合や、人物や物体の動きが所定速度よりも早い場合等である。本実施の形態では、生成する新たな画像のフレーム間で滑らかにするための補間処理について説明する。
本実施の形態では、ビデオカメラによって撮影した画像から新たに生成された画像の記録について説明する。
200 レンズ群
201 撮像素子
202 画像AD変換部
203,2203 画像信号処理部
204,2204 画像信号圧縮部
205 レンズ制御モジュール
206 姿勢検出部
207 外部入力部
208 マイクロフォン
209 音声AD変換部
210 音声信号圧縮部
211 画像信号伸張部
212 映像表示部
213 音声信号伸張部
214 音声出力部
215 出力I/F
300,1300,2300 制御部
301 クロック
302,2302 記憶部
311 レンズ制御部
312 撮像制御部
313 画像解析部
314 音声解析部
315,2315 カメラワーク情報抽出部
316,1316 構図情報計算部
317,2317 構図情報整形部
318,2318 多重化部
319,2319 再生部
Claims (6)
- 撮影された画像の撮影情報を抽出するカメラワーク情報抽出部と、
前記撮影情報と前記撮影された画像から、制約条件に基づいて、新たな画像を切り出すための切り出し枠を決定する構図情報計算部と、
前記撮影情報と前記切り出し枠とを前記撮影された画像の属性情報とする構図情報整形部と、
を備える画像生成装置。 - 前記制約条件は、切り出し枠のサイズに関する制約、画像の1コマの単位であるフレーム間の切り出し枠のサイズの変化量に関する制約、画像の1コマの単位であるフレームの構図に関する制約とフレーム間にまたがるカメラワークに関する制約のうち少なくとも1つを含む、
請求項1に記載の画像生成装置。 - 前記撮影情報は、
画像の撮影時の状態やカメラワークに関する情報、画像そのものが含む被写体や背景に関する情報と画像に付帯する音声に関する情報のうち少なくとも1つを含む、
請求項1に記載の画像生成装置。 - 画像を取得する撮像系と、
請求項1乃至3のいずれか1つに記載の画像生成装置と、
を備える撮影装置。 - 撮影された画像の撮影情報を抽出し、
前記撮影情報と前記撮影された画像から、制約条件に基づいて、新たな画像を切り出すための切り出し枠を決定し、
前記撮影情報と前記切り出し枠とを前記撮影された画像の属性情報とする、画像生成方法。 - 撮影された画像の撮影情報を抽出すること、
撮前記撮影情報と前記撮影された画像から、制約条件に基づいて、新たな画像を切り出すための切り出し枠を決定すること、
前記撮影情報と前記切り出し枠とを前記撮影された画像の属性情報とすること、をコンピュータに実行させるためのプログラム。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/761,054 US9848133B2 (en) | 2013-03-26 | 2014-02-26 | Image generation device, imaging device, image generation method, and program for generating a new image from a captured image |
JP2015507994A JP6314321B2 (ja) | 2013-03-26 | 2014-02-26 | 画像生成装置、撮影装置、画像生成方法及びプログラム |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013063448 | 2013-03-26 | ||
JP2013-063448 | 2013-03-26 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2014155963A1 true WO2014155963A1 (ja) | 2014-10-02 |
Family
ID=51622986
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2014/001006 WO2014155963A1 (ja) | 2013-03-26 | 2014-02-26 | 画像生成装置、撮影装置、画像生成方法及びプログラム |
Country Status (3)
Country | Link |
---|---|
US (1) | US9848133B2 (ja) |
JP (1) | JP6314321B2 (ja) |
WO (1) | WO2014155963A1 (ja) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6937477B2 (ja) * | 2017-04-24 | 2021-09-22 | パナソニックIpマネジメント株式会社 | 制御システム、シーン制御の実行方法、及び、プログラム |
JP7396919B2 (ja) * | 2020-02-12 | 2023-12-12 | シャープ株式会社 | 電子機器、撮像表示制御装置、撮像表示システム、撮像表示制御方法、および、プログラム |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH08255257A (ja) * | 1995-03-17 | 1996-10-01 | Fujitsu Social Sci Lab:Kk | 静止画像入力装置 |
JP2008205961A (ja) * | 2007-02-21 | 2008-09-04 | Canon Inc | 画像処理装置、その制御方法、及びプログラム |
JP2012217132A (ja) * | 2011-03-31 | 2012-11-08 | Sony Corp | 画像処理装置、画像処理方法、画像処理プログラムおよび撮像装置 |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100215586B1 (ko) | 1992-11-09 | 1999-08-16 | 모리시타 요이찌 | 다이제스트 화상 자동생성 장치 및 다이제스트 화상 자동생성 방법 |
JP3213105B2 (ja) | 1993-02-23 | 2001-10-02 | 日本電信電話株式会社 | 映像の構図自動決定処理方法 |
JP4366023B2 (ja) | 2001-03-16 | 2009-11-18 | インターナショナル・ビジネス・マシーンズ・コーポレーション | ビデオ・イメージの部分イメージ領域抽出方法、部分イメージ領域抽出システム、部分イメージ領域抽出のためのプログラム、抽出されたビデオ・イメージの配信方法およびコンテンツ作成方法 |
JP2007013479A (ja) | 2005-06-29 | 2007-01-18 | Matsushita Electric Ind Co Ltd | カメラワーク情報付与および評価装置 |
JP2007018198A (ja) | 2005-07-06 | 2007-01-25 | Sony Corp | リンク情報付きインデックス情報生成装置、タグ情報付き画像データ生成装置、リンク情報付きインデックス情報生成方法、タグ情報付き画像データ生成方法及びプログラム |
WO2007014216A2 (en) | 2005-07-22 | 2007-02-01 | Cernium Corporation | Directed attention digital video recordation |
US8848053B2 (en) | 2006-03-28 | 2014-09-30 | Objectvideo, Inc. | Automatic extraction of secondary video streams |
US20070250898A1 (en) | 2006-03-28 | 2007-10-25 | Object Video, Inc. | Automatic extraction of secondary video streams |
TW200924534A (en) | 2007-06-04 | 2009-06-01 | Objectvideo Inc | Intelligent video network protocol |
US8587658B2 (en) | 2007-06-08 | 2013-11-19 | Nikon Corporation | Imaging device, image display device, and program with intruding object detection |
JP5111088B2 (ja) | 2007-12-14 | 2012-12-26 | 三洋電機株式会社 | 撮像装置及び画像再生装置 |
JP2009182400A (ja) | 2008-01-29 | 2009-08-13 | Sony Corp | 画像処理装置、画像処理方法、画像処理方法のプログラム及び画像処理方法のプログラムを記録した記録媒体 |
CN102124727B (zh) * | 2008-03-20 | 2015-07-29 | 无线电技术研究学院有限公司 | 将视频图像适配到小屏幕尺寸的方法 |
JP2010062853A (ja) * | 2008-09-03 | 2010-03-18 | Fujifilm Corp | 情報処理装置、方法およびプログラム |
US20100074557A1 (en) | 2008-09-25 | 2010-03-25 | Sanyo Electric Co., Ltd. | Image Processing Device And Electronic Appliance |
JP5202211B2 (ja) | 2008-09-25 | 2013-06-05 | 三洋電機株式会社 | 画像処理装置及び電子機器 |
JP2010103972A (ja) | 2008-09-25 | 2010-05-06 | Sanyo Electric Co Ltd | 画像処理装置及び電子機器 |
JP2011078008A (ja) | 2009-10-01 | 2011-04-14 | Nippon Hoso Kyokai <Nhk> | コンテンツ共有装置、コンテンツ編集装置、コンテンツ共有プログラム、およびコンテンツ編集プログラム |
JP2011188210A (ja) * | 2010-03-08 | 2011-09-22 | Olympus Imaging Corp | 撮影機器及び撮影システム |
JP2012205037A (ja) | 2011-03-25 | 2012-10-22 | Olympus Imaging Corp | 画像処理装置および画像処理方法 |
-
2014
- 2014-02-26 JP JP2015507994A patent/JP6314321B2/ja active Active
- 2014-02-26 WO PCT/JP2014/001006 patent/WO2014155963A1/ja active Application Filing
- 2014-02-26 US US14/761,054 patent/US9848133B2/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH08255257A (ja) * | 1995-03-17 | 1996-10-01 | Fujitsu Social Sci Lab:Kk | 静止画像入力装置 |
JP2008205961A (ja) * | 2007-02-21 | 2008-09-04 | Canon Inc | 画像処理装置、その制御方法、及びプログラム |
JP2012217132A (ja) * | 2011-03-31 | 2012-11-08 | Sony Corp | 画像処理装置、画像処理方法、画像処理プログラムおよび撮像装置 |
Also Published As
Publication number | Publication date |
---|---|
JPWO2014155963A1 (ja) | 2017-02-16 |
US9848133B2 (en) | 2017-12-19 |
JP6314321B2 (ja) | 2018-04-25 |
US20150381903A1 (en) | 2015-12-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11902704B2 (en) | Apparatus and method for video-audio processing, and program for separating an object sound corresponding to a selected video object | |
JP5685732B2 (ja) | 映像抽出装置、プログラム及び記録媒体 | |
US8218033B2 (en) | Sound corrector, sound recording device, sound reproducing device, and sound correcting method | |
JP2009156888A (ja) | 音声補正装置及びそれを備えた撮像装置並びに音声補正方法 | |
WO2015098110A1 (ja) | 撮像装置、撮像システムおよび撮像方法 | |
JP6295442B2 (ja) | 画像生成装置、撮影装置、画像生成方法及びプログラム | |
JP6314321B2 (ja) | 画像生成装置、撮影装置、画像生成方法及びプログラム | |
JP2012151544A (ja) | 撮像装置及びプログラム | |
JP6295443B2 (ja) | 画像生成装置、撮影装置、画像生成方法及びプログラム | |
US20160172004A1 (en) | Video capturing apparatus | |
JP2020108047A (ja) | 画像処理装置、移動体デバイス、画像処理方法および画像処理プログラム | |
JP2014236276A (ja) | 画像処理装置、撮像装置および画像処理プログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14776329 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2015507994 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14761054 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 14776329 Country of ref document: EP Kind code of ref document: A1 |