Classifications

• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B20/00—Signal processing not specific to the method of recording or reproducing; Circuits therefor
  • G11B20/10—Digital recording or reproducing
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
  • H04N21/85—Assembly of content; Generation of multimedia applications
  • H04N21/854—Content authoring

Definitions

• This disclosure relates to a program, a method, an information processing device, and a system.
• Patent Document 1 describes a technology applicable to surveillance camera systems for specifying a tracking target before or during tracking of an object.
• the user selects the target (tracking target) they wish to enlarge from the extracted tracking target candidates, and obtains the desired enlarged image (zoomed image).
• Patent Document 1 one type of tracking target is extracted from one frame.
• one frame may contain multiple types of tracking targets.
• the purpose of this disclosure is to improve convenience when editing videos based on images captured by surveillance cameras, etc.
• a program to be executed by a computer having a processor and memory causes the processor to execute the following steps: acquiring a captured video; analyzing the acquired video using a plurality of pre-stored analytical models, each of which has been trained to detect an object; presenting to the user detection markers that indicate the timing at which the object was detected by the analytical models, by associating them with a timeline associated with the video; and generating an edited video from the captured video in response to user instructions based on the detection markers.
• This disclosure improves convenience when editing videos based on images captured by surveillance cameras, etc.
• FIG. 1 is a block diagram showing the overall configuration of a system.
• FIG. 2 is a block diagram showing the configuration of the terminal device shown in FIG. 1 .
• FIG. 2 is a diagram illustrating a functional configuration of a server.
• FIG. 2 is a diagram illustrating the data structure of a photographing device DB.
• FIG. 2 is a diagram illustrating the data structure of a detection log DB.
• 10 is a flowchart showing an operation when information detected by image analysis is recorded in a log.
• FIG. 10 is a schematic diagram illustrating an editing screen displayed on a display of a terminal device.
• FIG. 2 is a schematic diagram illustrating a part of an editing screen displayed on a display of a terminal device.
• 10A and 10B are schematic diagrams illustrating other examples of the editing screen displayed on the display of the terminal device.
• FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90.
• the system analyzes captured video using multiple image analysis models, presents objects that allow users to understand which model detected the target object, and creates an edited video based on the object designation.
• FIG. 1 is a block diagram showing an example of the overall configuration of a system 1.
• the system 1 shown in Fig. 1 includes, for example, a terminal device 10, a server 20, an image capturing device 31, a sensor 32, and a speaker 33.
• the terminal device 10, the server 20, and the speaker 33 are communicatively connected via, for example, a network 80.
• the system 1 includes two terminal devices 10, but the number of terminal devices 10 included in the system 1 is not limited to two.
• the terminal devices 10 are terminals carried by camera users.
• the number of terminal devices 10 included in the system 1 may be less than three, or may be three or more.
• a collection of multiple devices may be considered a single server.
• the method of allocating the multiple functions required to realize the server 20 of this embodiment to one or more pieces of hardware can be determined appropriately in consideration of the processing capacity of each piece of hardware and/or the specifications required of the server 20.
• the terminal device 10 shown in FIG. 1 may be, for example, a mobile terminal such as a smartphone or tablet, or a stationary PC (Personal Computer) or laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal.
• a mobile terminal such as a smartphone or tablet
• a stationary PC Personal Computer
• laptop PC Personal Computer
• HMD Head Mount Display
• the terminal device 10 includes a communication interface (IF) 12, an input device 13, an output device 14, memory 15, storage 16, and a processor 19.
• IF communication interface
• the communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with devices within the system 1, such as the server 20.
• the input device 13 is a device for accepting input operations from the user (for example, a touch panel, touchpad, pointing device such as a mouse, keyboard, etc.).
• the output device 14 is a device (display, speaker, etc.) for presenting information to the user.
• Memory 15 is used to temporarily store programs and data processed by programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory).
• DRAM Dynamic Random Access Memory
• Storage 16 is used to store data, and may be, for example, a flash memory or a hard disk drive (HDD).
• HDD hard disk drive
• the processor 19 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.
• the server 20 is realized, for example, by a computer connected to the network 80. As shown in FIG. 1, the server 20 includes a communication IF 22, an input/output IF 23, a memory 25, storage 26, and a processor 29.
• the communication IF 22 is an interface for inputting and outputting signals so that the server 20 can communicate with devices within the system 1, such as the terminal device 10.
• the input/output IF 23 functions as an interface with an input device for accepting input operations from the user and an output device for presenting information to the user.
• Memory 25 is used to temporarily store programs and data processed by programs, and is a volatile memory such as DRAM.
• Storage 26 is used to store data, and is, for example, a flash memory or HDD.
• the processor 29 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.
• the camera device 31 is a device that receives light using a light-receiving element and outputs the light as an image signal.
• the camera device 31 captures images at a frame rate that allows a series of movements to be recognized as a video.
• a frame rate that allows a series of movements to be recognized as a video is, for example, about 30 fps.
• the camera device 31 may be, for example, a camera that can capture a wide range of images in a 360-degree circle. In this case, the camera device 31 is realized, for example, by a camera with an ultra-wide-angle lens or a fisheye lens.
• the camera device 31 is installed at a location on the site where it can see the entire area without any obstructions. If a single camera device 31 cannot capture the entire site, multiple camera devices 31 are installed. When multiple camera devices 31 are installed, for example, the section of the site that will be photographed by each camera device 31 is set in advance. The camera device 31 outputs the captured image signal to the server 20.
• Fig. 2 is a block diagram showing an example configuration of the terminal device 10 shown in Fig. 1.
• the terminal device 10 shown in Fig. 2 is realized by a mobile terminal, a PC, or a wearable terminal.
• the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 161, a position information sensor 150, a storage unit 180, and a control unit 190.
• the blocks included in the terminal device 10 are electrically connected by, for example, a bus or the like.
• the communication unit 120 performs processes such as modulation and demodulation to enable the terminal device 10 to communicate with other devices.
• the communication unit 120 performs transmission processing on signals generated by the control unit 190 and transmits them to the outside (e.g., server 20).
• the communication unit 120 performs reception processing on signals received from the outside and outputs them to the control unit 190.
• the input device 13 is a device through which the user operating the terminal device 10 inputs instructions or information.
• the input device 13 is realized, for example, by a touch-sensitive device 131, which inputs instructions by touching the operation surface. If the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, keyboard, mouse, etc.
• the input device 13 converts instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190.
• the input device 13 may also include, for example, a receiving port that receives electrical signals input from an external input device.
• the output device 14 is a device for presenting information to a user operating the terminal device 10.
• the output device 14 is realized, for example, by a display 141.
• the display 141 displays data according to the control of the control unit 190.
• the display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.
• the audio processing unit 17 performs digital-to-analog conversion of audio signals.
• the audio processing unit 17 converts the signal provided by the microphone 171 into a digital signal and provides the converted signal to the control unit 190.
• the audio processing unit 17 also provides the audio signal to the speaker 172.
• the audio processing unit 17 is realized, for example, by a processor for audio processing.
• the microphone 171 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17.
• the speaker 172 converts the audio signal provided by the audio processing unit 17 into audio and outputs the audio externally from the terminal device 10.
• Camera 161 is a device that receives light using a light-receiving element and outputs it as an image signal.
• the location information sensor 150 is a sensor that detects the location of the terminal device 10, and is, for example, a GPS (Global Positioning System) module.
• a GPS module is a receiving device used in a satellite positioning system.
• a satellite positioning system receives signals from at least three or four satellites, and detects the current location of the terminal device 10 equipped with a GPS module based on the received signals.
• the location information sensor 150 may also detect the current location of the terminal device 10 from the location of the wireless base station to which the terminal device 10 is connected.
• the storage unit 180 is realized, for example, by the memory 15 and storage 16, and stores data and programs used by the terminal device 10.
• the storage unit 180 stores, for example, user information 181.
• User information 181 includes, for example, information about the user who uses the terminal device 10.
• Information about the user includes, for example, information to identify the user, the user's name, age, address, date of birth, telephone number, email address, etc.
• the control unit 190 is realized when the processor 19 reads a program stored in the storage unit 180 and executes the instructions contained in the program.
• the control unit 190 controls the operation of the terminal device 10.
• the control unit 190 fulfills the functions of an operation reception unit 191, a transmission/reception unit 192, and a presentation control unit 193.
• the operation reception unit 191 performs processing to receive instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives information based on instructions input from the touch-sensitive device 131 or the like. Instructions input via the touch-sensitive device 131 or the like are, for example, editing instructions from the user.
• the operation reception unit 191 also receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 17. The operation reception unit 191 acquires instructions from the user, for example, by analyzing the received voice signal and extracting predetermined nouns.
• the transmission/reception unit 192 performs processing to enable the terminal device 10 to send and receive data to and from external devices such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmission/reception unit 192 sends editing instructions input by the user to the server 20. The transmission/reception unit 192 also receives information about the user from the server 20.
• the presentation control unit 193 controls the output device 14 to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 causes the information transmitted from the server 20 to be displayed on the display 141. The presentation control unit 193 also causes the information transmitted from the server 20 to be output from the speaker 172.
• Functional configuration of the server> 3 is a diagram showing an example of the functional configuration of the server 20.
• the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.
• the communication unit 201 performs processing that enables the server 20 to communicate with external devices.
• the storage unit 202 includes, for example, an imaging device database (DB) 2021 and a detection log database (DB) 2022.
• DB imaging device database
• DB detection log database
• the camera DB2021 is a database for storing information about cameras installed for photography. Details will be provided below.
• Detection log DB2022 is a database for storing information about objects detected through image analysis. Details will be provided below.
• the first trained model 2023 is a model generated by having a machine learning model perform machine learning in accordance with a model learning program.
• the first trained model 2023 is, for example, a parameterized composite function formed by combining multiple functions that performs predetermined inference based on input data.
• the parameterized composite function is defined by a combination of multiple adjustable functions and parameters.
• the trained model in this embodiment may be any parameterized composite function that meets the above requirements. The same applies to the second trained model 2024 and the third trained model 2025.
• the parameterized composite function is defined as a combination of, for example, linear relationships between each layer using weight matrices, non-linear relationships (or linear relationships) using activation functions in each layer, and biases.
• the weight matrices and biases are called parameters of the multi-layer network.
• the form of the parameterized composite function as a function changes depending on how the parameters are selected. In a multi-layer network, by appropriately setting the constituent parameters, it is possible to define a function that can output desirable results from the output layer. The same is true for the second trained model 2024 and the third trained model 2025.
• a deep neural network which is a multi-layered neural network that is the subject of deep learning
• a convolutional neural network CNN
• targets images may also be used.
• the first trained model 2023 is a model that learns to output information about a human being based on an input image when the image is input.
• the first trained model 2023 may be, for example, different trained models that are trained with different learning data depending on the information to be output.
• the first trained model 2023 learns to output information about human attributes such as gender, age, race, and occupational status. Specifically, in this embodiment, the first trained model 2023 identifies human attributes based on information about the physique, face, hairstyle, clothing, and belongings of the human included in the image. Also, for example, the first trained model 2023 may be trained to output information about a specific person when an image is input. Specifically, in this embodiment, the first trained model 2023 may identify a specific person based on information about the face, hairstyle, and physique of the specific person included in the image. Also, for example, the first trained model 2023 may be trained to output information about human body parts when an image is input.
• human attributes such as gender, age, race, and occupational status. Specifically, in this embodiment, the first trained model 2023 identifies human attributes based on information about the physique, face, hairstyle, clothing, and belongings of the human included in the image. Also, for example, the first trained model 2023 may be trained to output information about a specific person when an image is input.
• the first trained model 2023 may identify human body parts such as hands or face based on the skeletal arrangement of the human included in the image. At this time, the first trained model 2023 identifies human body parts included in the image using a method such as open pose, for example.
• the learning data may, for example, be an image of a human being as input data, and the judgments about attributes, specific people, and human body parts contained in this input data may be used as correct output data. In this case, the learning data may not need to include correct output data.
• the second trained model 2024 is a model that has been trained to output information about an object based on an input image when the image is input.
• the object here refers to an animal or plant, clothing, machinery, a vehicle, a craft, interior design, a building, etc.
• the second trained model 2024 may be, for example, a separate trained model that is trained with separate learning data depending on the information to be output.
• the second trained model 2024 learns to output information about an object when an image is input. Specifically, in this embodiment, the second trained model 2024 identifies the object based on information such as the size, shape, material, color, and accessories of the object contained in the image.
• the learning data may, for example, be an image of an object as input data, and a judgment about the object represented in this input data may be the correct output data. In this case, the learning data may not include the correct output data.
• the third trained model 2025 is a model that learns to output information about a movement based on an input image when the image is input.
• the third trained model 2025 may be, for example, separate trained models that are trained with separate learning data depending on the information to be output.
• the third trained model 2025 is trained to output information related to shopping when an image is input. Specifically, in this embodiment, the third trained model 2025 is trained to output that a shopping-related activity has been performed when an image including the following content is input: A customer taking out money or a card near a cash register in a retail store, an action spanning multiple frames. A customer picking up an item in a retail store, an action spanning multiple frames.
• the learning data may, for example, use images of the interior of a retail store as input data, and the judgment regarding the actions represented in this input data may be used as correct output data. In this case, the learning data may not need to include correct output data.
• the control unit 203 is realized when the processor 29 reads a program stored in the storage unit 202 and executes the instructions contained in the program. By operating in accordance with the program, the control unit 203 performs the functions shown as a reception control module 2031, a transmission control module 2032, an image analysis module 2034, an image editing module 2035, and a presentation module 2036.
• the reception control module 2031 controls the process by which the server 20 receives signals from external devices in accordance with a communication protocol.
• the transmission control module 2032 controls the process by which the server 20 transmits signals to external devices in accordance with a communication protocol.
• the image analysis module 2034 analyzes images captured by the imaging device 31 and outputs information about objects at the site.
• the image analysis module 2034 inputs the images to the first learned model 2023, the second learned model 2024, and the third learned model 2025. More specifically, the image analysis module 2034 analyzes the image using, for example, the first learned model 2023, and after the analysis by the first learned model 2023 is completed, analyzes the image using the second learned model 2024. After the analysis by the second learned model 2024 is completed, the image analysis module 2034 analyzes the image using the third learned model 2025.
• the first learned model 2023, the second learned model 2024, and the third learned model 2025 output information about objects corresponding to each model in accordance with the input image.
• the image analysis module 2034 outputs information output from the first trained model 2023, the second trained model 2024, and the third trained model 2025 to the presentation module 2036.
• the image analysis module 2034 does not necessarily have to use the first trained model 2023, the second trained model 2024, and the third trained model 2025, and may analyze the image using a trained model specified by the user.
• the image analysis module 2034 may input an image with a limited analysis range into the trained model.
• the analysis range may be specified by a user, for example.
• time may be added to the conditions for analyzing an image. For example, the image analysis module 2034 analyzes an image only if an object continues to be detected in the analysis range for a certain period of time or more.
• the image editing module 2035 receives editing operations from the user and edits the video analyzed by the image analysis module 2034. Specifically, for example, the image editing module 2035 edits the video from the perspectives of time, objects, image range, etc.
• the image editing module 2035 may edit the video in response to point-by-point operations by the user, or may edit the video automatically based on rules set by the user.
• the presentation module 2036 presents information about objects on-site to the user via the terminal device 10.
• the presentation module 2036 may present information to the user based on instructions from the user.
• FIG. 4 and 5 are diagrams showing the data structure of the database stored in the server 20. Note that Fig. 4 and Fig. 5 are merely examples and do not exclude data not shown.
• FIG 4 is a diagram showing the data structure of the imaging device DB 2021.
• each record in the imaging device DB 2021 includes, for example, an item "imaging device ID,” an item “coordinates,” an item “address,” an item “model,” an item “installation date,” an item “removal date,” an item “operation status,” an item “latest analysis,” an item “current analysis status,” an item “analysis frequency,” and an item “photographed video.”
• the "Photography Device ID” item indicates identification information for identifying the photography device 31.
• the "Coordinates” item indicates the installation location of the camera device 31. Specifically, the “Coordinates” item indicates the latitude and longitude of the installation location of the camera device 31. Note that the installation location of the camera device 31 may be determined by indicators other than coordinates.
• the "Address" item indicates the address where the imaging device 31 is installed.
• the "Model” item indicates the model of the imaging device 31.
• the "Installation Date” item indicates the date on which the imaging device 31 was installed.
• the "Removal Date” item indicates the date the imaging device 31 was removed.
• the "Operation Status” item indicates the operation status of the imaging device 31. Specifically, the “Operation Status” item is displayed as “In Operation,” “Stopped,” “Out of Order,” or “Removed” depending on the status of the imaging device 31.
• the "Latest Analysis” item indicates the time point of the most recent image that the server 20 has analyzed. Specifically, the server 20 analyzes images received from the imaging device 31 at a predetermined timing, and the "Latest Analysis” item indicates the time point of the most recent analyzed image. In other words, images up to the time point indicated by the "Latest Analysis” item in Figure 4 (for example, 2023/10/22 23:59:59) have been analyzed.
• the item "Current Analysis Status” indicates the time division of the image currently being analyzed by the server 20.
• the time division can be set arbitrarily. For example, the division is as follows: - Daily (00:00:00 to 23:59:59) Morning (00:00:00-9:59:59), afternoon (10:00:00-17:59:59), evening (18:00:00-23:59:59)
• the "Analysis Frequency” item indicates how often the server 20 analyzes images.
• the "Video” item indicates a video captured by the camera device 31. Specifically, the video is video data captured by the camera device 31. The video may also store reference information to a video data file located elsewhere.
• FIG. 5 is a diagram showing the data structure of the detection log DB2022.
• each record in the detection log DB2022 includes, for example, the items "camera ID,” “detection model,” “detection category,” “detection frame,” and “image area.”
• the trained model When an object is detected by the trained model, a new record is created in the detection log DB2022, and each piece of information is stored in the corresponding item.
• the "Photography Device ID” item indicates identification information for identifying the photography device 31.
• the "Detection Model” item indicates the trained model that detected the object.
• the detection model used may be the first trained model 2023, the second trained model 2024, the third trained model 2025, etc., stored in the memory unit 202.
• the "Detection Category” item indicates the category of the detected object. Specifically, for example, the category is the classification of the object detected by the trained model. For example, the object detected by the first trained model 2023 is classified as an "adult male” among humans.
• the "Detection Frame” item indicates the frame of the image in which the object was detected.
• the "Image Area” item indicates the area of the object on the image. Specifically, for example, if the image is divided into a mesh, the "Image Area” item indicates the mesh in which the object appears in the image.
• the camera device 31 is attached, for example, to the ceiling overlooking the equipment within the store.
• the equipment is, for example, items and locations used for purchasing or selling.
• the equipment includes, for example, shelves, entrances and exits, cash registers, and shopping carts.
• the camera device 31 captures images of the store at a frame rate that allows a series of movements to be recognized as a video.
• a frame rate that allows a series of movements to be recognized as a video is, for example, around 30 fps.
• the camera device 31 transmits image data to the server 20.
• the transmission frequency is a fixed cycle based on the data volume of the unsent video.
• the image capture device 31 transmits the unsent video to the server 20 every time the data volume of the unsent video reaches a predetermined volume.
• the transmission frequency may also be a fixed cycle based on the time of the unsent video.
• the image capture device 31 transmits the unsent video to the server 20 every time a predetermined time has elapsed since the most recent transmission.
• the image capture device 31 may also immediately transmit the captured video to the server 20.
• Figure 6 is a flowchart showing the operations involved in recording information detected by image analysis in a log.
• the control unit 203 analyzes the image captured by the imaging device 31 using the first trained model 2023 recorded in the memory unit 202.
• the image analysis module 2034 inputs image data transmitted from the imaging device 31 into the first trained model 2023.
• the frequency with which the image analysis module 2034 analyzes the image is periodic.
• the image analysis module 2034 may analyze the image, for example, weekly, every few days, daily, or every few hours.
• the image analysis may be performed, for example, at 9:00 p.m. every Sunday.
• the image analysis module 2034 may also analyze the image immediately after the server 20 receives the image. In other words, the frequency with which the image analysis module 2034 analyzes the image can be set arbitrarily.
• the first trained model 2023 outputs information related to the detected object.
• step S62 the control unit 203 determines whether or not there is an object in the analyzed image that corresponds to the first trained model 2023. If there is a corresponding object, the control unit 203 transitions the processing to step S63. On the other hand, if there is no corresponding object, the control unit 203 transitions the processing to step S64.
• step S63 the control unit 203 records information related to the object detected in step S62 in the detection log DB 2022.
• the recorded information includes, for example, the imaging device ID, detection model, detection category, detection frame, and image area.
• step S64 if there are any trained models remaining that have not been used for detection, the control unit 203 repeats the processes of steps S61, S62, and S63 using the unused trained models. If there are no trained models remaining that have not been used for detection, the control unit 203 ends the process. When the control unit 203 has finished the process, the control unit 203 may notify the display 141 of the terminal device 10 that the control unit 203 has finished the process.
• the user operates the terminal device 10 and accesses the server 20 after user authentication.
• the server 20 sends information about the editing screen to the terminal device 10.
• the editing screen includes information from the detection log DB 2022 that has been updated since the previous access.
• the user edits the video on the editing screen.
• Figure 7 is a schematic diagram showing an editing screen displayed on the display 141 of the terminal device 10.
• Display area 71 is an area where video is displayed in streaming format.
• Marks 721, 722, and 723 are borders that surround objects detected in the video.
• the style of the borders may differ for each trained model or detection category.
• the style of the borders may include, for example, color, thickness, shape, etc., or a combination of these. Even if an object of only one detection model or only one detection category is detected in the video, a mark may be attached to the object.
• Number 73 indicates the time point in the video that is being displayed.
• Icon group 74 is an icon for operating the video. Icon group 74 is not limited to the three icons shown in Figure 7.
• the timeline 75 is a time axis for displaying the time associated with the detection marker 78 of each box 77 on a chronological order.
• the timeline 75 is divided into sections at regular time points, for example.
• the playback marker 76 is a marker on the timeline 75 that indicates the time point of the video currently being played.
• the box 77 is a box that indicates the category of the detected object.
• the box 77 may also indicate information that can identify the trained model of the detected object.
• the detection marker 78 is a marker on the timeline 75 that indicates the time at which the object was detected. In the example of Figure 7, the dog was detected between 8:00 and 9:00.
• Icon 79 is an icon for editing a video.
• icon 79 is an icon for dividing a video into certain time segments and then cutting out the video.
• the user may determine the time of the video to be cut out by specifying detection marker 78. That is, image editing module 2035 cuts out the video for the time period corresponding to detection marker 78.
• Image editing module 2035 is not limited to matching the time period of the video to be cut out with detection marker 78, and may add a predetermined time width to detection marker 78.
• image editing module 2035 may add a predetermined time width before the start point of detection marker 78, after the end point of detection marker 78, or both.
• the user may determine the time of the video to be cut out by, for example, specifying a time width on the timeline 75.
• the image editing module 2035 receives a tap or a drag at two points on the timeline 75 from the user, and cuts out the video of the received time width.
• the image editing module 2035 saves the edited video file, for example, in memory 25 or storage 26, etc.
• Figure 8 is a schematic diagram showing a portion of the editing screen displayed on the display 141 of the terminal device 10.
• the user may, for example, determine the time period of the video to be cut out by specifying two or more detection markers.
• the user may determine the time period of the video to be cut out by specifying two detection markers that are separated from each other on the timeline.
• the image editing module 2035 may cut out the video in the time periods corresponding to each detection marker specified by the user and combine the videos.
• the two specified detection markers may belong to the same category or different categories.
• the user may determine the time of the video to be cut out by specifying two overlapping detection markers on the timeline.
• the image editing module 2035 extracts the portion where the two detection markers specified by the user overlap.
• the image editing module 2035 extracts the portion corresponding to at least one detection marker.
• Icon 79 in Figure 7 is not limited to the icon for cutting out shown in Figure 7.
• icon 79 may be an icon for editing a video.
• icon 79 may be, for example, an icon for adjusting the brightness of a video, or an icon for applying a predetermined effect to a video.
• the image editing module 2035 may accept an operation on icon 79 from the user and edit the video.
• the user may also arbitrarily set rules for editing videos.
• the user may set rules for automatic editing based on an object as a condition. For example, the user may set rules for automatically determining the time of the video to be cut out by specifying a child detection marker 78 and a toy detection marker 78. As a result, when a specific object is detected, a video including that object is automatically generated.
• the user may also set rules for automatic editing based on time as a condition. For example, the user may set a rule to automatically increase the brightness of videos shot at night. As a result, videos will be automatically generated during a specific time period.
• the image editing module 2035 may automatically edit videos according to any video editing rules set by the user.
• Icon 710 in Figure 7 is an icon for downloading the video edited by icon 79.
• the transmission control module 2032 transmits the edited video file from memory 25, storage 26, etc. to the terminal device 10.
• the terminal device 10 saves the downloaded video file.
• the control unit 203 acquires the captured video.
• the control unit 203 analyzes the video using a plurality of pre-stored analysis models, each of which has been learned for a different object.
• the control unit 203 presents the user with a detection marker 78, which indicates the timing at which the object was detected by the analysis model, in association with a timeline 75 associated with the video.
• the control unit 203 generates an edited video from the captured video in response to user instructions based on the detection marker 78. This allows the user to intuitively and easily grasp the situation at the scene where the video was captured, via a user interface in which the appearance times of objects for each model are visually organized on the timeline 75. This also allows the user to intuitively and easily edit the video via a user interface for efficiently processing objects.
• the system according to this embodiment improves convenience when editing videos based on images captured by surveillance cameras, etc.
• control unit 203 receives a specification for the detection marker 78 in the generating step, and generates an edited video for the time period corresponding to the detection marker 78, or for a time period that has a predetermined range around that time period. This allows the user to edit the video more intuitively and simply via a user interface for efficiently processing objects. This also allows the user to not miss any signs before an object appears or any effects after the object has disappeared.
• control unit 203 receives specifications for multiple detection markers 78 in the generating step, and generates an edited video of a combined time period by joining together time periods corresponding to the detection markers 78, or a combined time period by joining together time periods with a predetermined width therebetween. This allows the user to edit the video more intuitively and simply via a user interface for efficiently processing objects. This also allows the user to not miss any signs before an object appears or any effects after the object has disappeared.
• control unit 203 if there is a time period in which detection markers 78 detected by different analysis models overlap in the generating step, the control unit 203 generates an edited video for the overlapping time period or for a time period with a predetermined width around that time period. This allows the user to edit the video more intuitively and simply via a user interface for efficiently processing objects. This also allows the user to not miss any signs before an object appears or any effects after the object has disappeared.
• the control unit 203 if there is a time period in which detection markers 78 from different analysis models overlap in the generating step, the control unit 203 generates an edited video for the time period corresponding to at least one of the detection markers 78, or for a time period with a predetermined range around that time period. This allows the user to edit the video more intuitively and simply via a user interface for efficiently processing objects. This also allows the user to not miss any signs before an object appears or any effects after the object has disappeared.
• control unit 203 accepts the specification of time periods on the timeline 75 in the generating step, and generates an edited video for the specified time periods or a combined time period that joins together these time periods. This allows the user to edit the video more intuitively and simply via a user interface for efficiently processing objects. This also allows the user to edit the video while referring to the detection markers 78, without being limited by the detection markers 78.
• the control unit 203 receives instructions from the user specifying rules for generating an edited video based on the detected markers 78. This allows the user to edit the video more intuitively and simply via a user interface for efficiently processing objects. This also saves the user the trouble of having to input frequently occurring editing patterns.
• control unit 203 analyzes the video at any frequency using the analysis model in the analysis step.
• the frequency of image analysis by the image analysis module 2034 can be set arbitrarily, so the device according to this embodiment does not necessarily require high performance.
• the control unit 203 simultaneously presents marks 721-723 representing those objects to the user. This allows the user to intuitively and easily grasp the situation of the scene photographed by the marks 721-723 attached to each object, even if multiple objects are detected in the same frame.
• control unit 203 presents the marks 721 to 723 in different forms for each analysis model in the presentation step. This allows the user to intuitively and easily grasp the situation of the captured site from the marks 721 to 723, which have different forms for each analysis model.
• the image capturing device 31 transmits video to the server 20.
• the server 20 may be realized by a general information processing device such as a PC.
• a storage medium storing video in the image capturing device 31 is removed and connected to an input port of the information processing device, thereby inputting the information stored in the storage medium to the information processing device.
• the control unit of the information processing device performs processing similar to that of the image analysis module 2034 of the server 20, for example, analyzing images captured by the image capturing device 31 and outputting information about objects at the site.
• the control unit of the information processing device also performs processing similar to that of the image editing module 2035 of the server 20, for example, accepting editing instructions from a user and generating an edited video based on the accepted editing instructions.
• the edited video is downloaded from memory 25, storage 26, etc. to the terminal device 10 and saved there.
• the user may also view the edited video on the server 20 without downloading it.
• the image analysis module 2034 may limit the analysis range of an image. There may be two or more analysis ranges for one image. For example, the ranges analyzed for a video taken of a retail store may be two locations: the entrance and the exit. Furthermore, if objects are detected simultaneously in two or more analysis ranges, or if objects are detected within a certain time interval, the video for the time at which they were detected may be extracted.
• a video of a time period equal to the detected marker plus a predetermined time width may be cut out.
• the video to be created is not limited to this.
• the image editing module 2035 may cut out a video of a time period consisting of only that predetermined time width.
• the videos referenced are not limited to those captured by one camera device 31.
• the image analysis module 2034 may analyze videos captured by two or more camera devices 31.
• the image editing module 2035 may generate an edited video based on videos captured by two or more camera devices 31. Specifically, for example, the image analysis module 2034 analyzes multiple videos captured by multiple camera devices 31 installed to capture the same site from different angles. The image editing module 2035 generates an edited video based on the analysis results.
• FIG. 9 is a schematic diagram showing another example of an editing screen displayed on the display 141 of the terminal device 10. Note that while FIG. 9 shows a case where one object is detected, multiple objects may be detected. In other words, multiple trained models may be used in the analysis.
• the editing screen in FIG. 9 includes, for example, multiple videos and one timeline that collectively corresponds to those videos.
• detection marker 78 indicates that an object has been detected in the video.
• presentation module 2036 may display detection markers for each video from each camera under the same object. By manipulating multiple detection markers under the same object, a user can extract videos based on, for example, the time at which the same object was detected in multiple videos.
• the image editing module 2035 may, for example, provide the user with a choice of videos to edit. Specifically, for example, the image editing module 2035 may accept from the user a selection of videos to be included in the edited video. For example, when the user selects one or more detection markers under the same object, the image editing module 2035 sets the video corresponding to the selected detection markers as the subject of editing. The user may also select a video in which no detection markers appear under the same object.
• the image editing module 2035 edits the selected video in the same way as if one video and one timeline corresponding to that video were displayed on the editing screen.
• the image editing module 2035 edits the selected videos after specifying how the multiple videos should be played, based on the user's specifications. In other words, the user specifies whether the selected multiple videos should be played in sequence or in parallel.
• the order of playback is determined automatically or by user operation.
• the automatically determined order of playback may be determined, for example, based on time, the proportion of the screen occupied, etc.
• the image editing module 2035 determines the order of playback by taking into consideration, for example, the length of time indicated by the detection marker in the video, the average proportion of the area of the object that occupies the screen throughout the video, etc.
• the image editing module 2035 may suggest the automatically determined order to the user as an aid in making the decision.
• the primary-subordinate relationship between the videos is determined automatically or by user operation.
• An automatically determined primary-subordinate relationship may be determined based on, for example, time, the proportion of the screen occupied, etc.
• the image editing module 2035 determines the primary-subordinate relationship by taking into consideration, for example, the length of time indicated by the detection marker in the video, the average proportion of the area of the object that is occupied on the screen throughout the video, etc.
• the image editing module 2035 may suggest the automatically determined primary-subordinate relationship to the user as an aid in the decision.
• the multiple videos may be played in, for example, divided video playback areas.
• the image editing module 2035 divides the video playback area so that multiple videos can be played in the video playback area.
• the area division is determined automatically or in response to a user operation.
• the automatically determined division may be determined based on, for example, time, the proportion of the screen occupied, etc.
• the image editing module 2035 determines the division taking into consideration, for example, the length of time indicated by the detection marker in the video, the average proportion of the area of the object that occupies the screen throughout the video, etc.
• the image editing module 2035 may suggest the automatically determined division to the user as an aid in the decision. Even if there are differences between the multiple videos in terms of time, the proportion of the screen occupied, etc., the image editing module 2035 may divide the video playback area evenly in response to a user operation.
• the presentation module 2036 may display output information of a predetermined IoT device on a timeline.
• the reception control module 2031 receives information measured by an IoT device attached to on-site equipment at a predetermined interval.
• the presentation module 2036 presents a detection marker based on the information transmitted from the IoT device on the timeline of the editing screen. For example, if the acquired information satisfies predetermined requirements, the presentation module 2036 presents the detection marker on the timeline of the editing screen.
• the presentation module 2036 when the reception control module 2031 receives information indicating a temperature above a certain level from a temperature sensor attached to equipment displayed on an image, the presentation module 2036 presents a detection marker on a timeline indicating the time during which that temperature was reached.
• the presentation module 2036 may present the detection marker so that the detection marker indicates each temperature range in a different manner. For example, an orange detection marker may indicate a temperature between 30 degrees Celsius and 40 degrees Celsius.
• the reception control module 2031 receives information about an acceleration equal to or greater than a certain level from an acceleration sensor attached to equipment displayed on an image
• the presentation module 2036 presents a detection marker on a timeline indicating the time during which that acceleration occurred.
• the presentation module 2036 may present the detection marker so that the detection marker indicates the acceleration range in a different manner for each acceleration range. For example, a blue detection marker may indicate an acceleration of 1 G or greater but less than 2 G.
• Basic Computer Hardware Configuration> 10 is a block diagram showing the basic hardware configuration of a computer 100.
• the computer 100 includes at least a processor 101, a main memory device 102, an auxiliary memory device 103, and a communication IF (interface) 109. These components are electrically connected to one another by a bus.
• the processor 101 is hardware that executes a set of instructions written in a program.
• the processor 101 is composed of an arithmetic unit, registers, peripheral circuits, etc.
• the main memory device 102 is used to temporarily store programs and data processed by the programs.
• it is a volatile memory such as DRAM (Dynamic Random Access Memory).
• the auxiliary storage device 103 is a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.
• the communication IF 109 is an interface for inputting and outputting signals for communicating with other computers over a network using wired or wireless communication standards.
• Networks are made up of the Internet, LANs, various mobile communication systems constructed using wireless base stations, etc.
• networks include 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the Internet via designated access points (such as Wi-Fi (registered trademark)).
• communication protocols include Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc.
• networks also include those that are directly connected using USB (Universal Serial Bus) cables, etc.
• the computer 100 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 100 and connecting them via a network.
• the concept of computer 100 includes not only a computer 100 housed in a single housing or case, but also a virtualized computer system.
• the computer includes at least the functional units of a control unit, a storage unit, and a communication unit.
• Functional units provided in the computer 100 can also be realized by distributing all or part of each functional unit across multiple computers 100 interconnected via a network.
• the concept of computer 100 includes not only a single computer 100 but also a virtualized computer system.
• the control unit is realized when the processor 101 reads various programs stored in the auxiliary storage device 103, expands them into the main storage device 102, and executes processing in accordance with those programs.
• the control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.
• the storage unit is realized by the main storage device 102 and the auxiliary storage device 103.
• the storage unit stores data, various programs, and various databases.
• the processor 101 can allocate a storage area corresponding to the storage unit in the main storage device 102 or the auxiliary storage device 103 in accordance with the programs.
• the control unit can cause the processor 101 to add, update, and delete data stored in the storage unit in accordance with the various programs.
• a database refers to a relational database, which manages data sets called tables, which are structured by rows and columns, by relating them to each other.
• a table is called a table
• a column in a table is called a column
• a row in a table is called a record.
• relationships between tables can be set and associated.
• each table has a column set as a key for uniquely identifying a record, but setting a key to a column is not essential.
• the control unit can cause the processor 101 to add, delete, or update records in a specific table stored in the storage unit according to various programs.
• the communication unit is realized by the communication IF 109.
• the communication unit realizes the function of communicating with other computers 100 via a network.
• the communication unit can receive information sent from other computers 100 and input it to the control unit.
• the control unit can cause the processor 101 to execute information processing on the received information in accordance with various programs.
• the communication unit can also transmit information output from the control unit to other computers 100.
• circuitry or processing circuitry including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and/or combinations thereof, programmed to perform the described functions.
• a processor includes transistors and other circuits and is considered to be circuitry or processing circuitry.
• a processor may also be a programmed processor that executes programs stored in memory.
• a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and/or processor.
• Appendix 1 A program for operating a computer having a processor 29 and a memory 25, the program causing the processor to execute the steps of acquiring a filmed video, analyzing the acquired video using a plurality of pre-stored analysis models, each of which has been trained to detect an object, presenting to the user a detection marker 78 that indicates the timing at which the object was detected by the analysis model, in association with a timeline 75 associated with the video, and generating an edited video from the filmed video in response to instructions from the user based on the detection marker.
• (Appendix 2) The program described in (Appendix 1) in which, in the generating step, a specification for a detection marker is accepted, and an edited video is generated for a time period corresponding to the detection marker or for a time period with a predetermined range around that time period.
• (Appendix 3) A program described in (Appendix 1) or (Appendix 2), in which, in the generating step, specifications for multiple detection markers are accepted and an edited video of a combined time period is generated by connecting time periods corresponding to the detection markers, or a combined time period is generated by connecting time periods with a predetermined width within the time period.
• (Appendix 4) A program described in any one of (Appendix 1) to (Appendix 3), in which, in the generating step, if there is a time period in which detected markers by different analysis models overlap, an edited video is generated for the overlapping time period or for a time period with a predetermined width within that time period.
• (Appendix 5) A program described in any of (Appendix 1) to (Appendix 4), in which, in the generation step, if there is a time period in which detection markers from different analysis models overlap, an edited video is generated for a time period corresponding to at least one of the detection markers, or for a time period with a predetermined range within that time period.
• (Appendix 6) A program described in any of (Appendix 1) to (Appendix 5), in which, in the generating step, a specification of a time period on a timeline is accepted and an edited video of the specified time period or a combined time period created by connecting these time periods is generated.
• (Appendix 7) The program according to any one of (Appendix 1) to (Appendix 6), wherein in the generating step, the user provides rules for generating the edited video based on the detected markers as instructions.
• (Appendix 8) A program described in any one of (Appendix 1) to (Appendix 7), wherein in the analyzing step, the video is analyzed at any frequency using an analytical model.
• (Appendix 9) A program described in any one of (Appendix 1) to (Appendix 8), wherein in the presentation step, if objects detected by different analysis models are in the same frame, marks representing the objects are presented to the user simultaneously.
• (Appendix 10) A program according to (Appendix 9), wherein in the presenting step, the mark has a different form for each analytical model.
• (Appendix 11) A method executed by a computer having a processor and a memory, wherein the processor executes all of the steps executed in the invention according to any one of (Appendix 1) to (Appendix 10).
• Appendix 12 An information processing device comprising a control unit 203 and a memory unit 202, wherein the control unit executes all steps executed in any of the inventions according to (Supplementary Note 1) to (Supplementary Note 10).
• Appendix 13 A system comprising means for executing all steps performed in any of the inventions according to (Appendix 1) to (Appendix 10).
• Input/output IF 25...Memory 26...Storage 29...Processor 31...Photographing device 80...Network 100...Server (computer) 101... Processor 102... Memory (main storage device) 103...Storage (auxiliary storage device) 109...Communication IF

Landscapes

• Engineering & Computer Science (AREA)
• Signal Processing (AREA)
• Multimedia (AREA)
• Computer Security & Cryptography (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
• Closed-Circuit Television Systems (AREA)
• Signal Processing For Digital Recording And Reproducing (AREA)
• Management Or Editing Of Information On Record Carriers (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=96545334

Family Applications (1)

Country Status (2)

Citations (2)

• 2024
  • 2024-01-23 JP JP2024008306A patent/JP7747423B2/ja active Active
• 2025
  • 2025-01-20 WO PCT/JP2025/001622 patent/WO2025159057A1/ja active Pending
  • 2025-09-16 JP JP2025153619A patent/JP2025178330A/ja active Pending

Patent Citations (2)

Also Published As

Similar Documents

Legal Events