JP2023180871A - Recording device, method for controlling recording device, and program - Google Patents

Abstract

To record a unit suitable for a subject in annotation information.SOLUTION: A recording device has: detection means that detects a subject from an image; measurement means that measures an extent and the like from the image in a common unit; selection means that selects a field of an industry recorded by a user; determination means that determines numerical values and units for all of subject information output from the detection means, a measurement result output from the measurement means, and industry information selected by the selection means; and recording means that records the numerical values and units, measurement result, and industry information output from the determination means together in annotation information. The recording means records them in the common unit in standard annotation information, and records them in a unit according to the industry in annotation information for the industry.SELECTED DRAWING: Figure 4

Description

The present invention relates to a recording device that processes images captured by a digital camera or the like.

Conventionally, when a certain subject is photographed, data related to the subject is saved as separate data from the image. For example, when the subject is a room, the size of the room corresponds to data related to the subject. However, as time passes, it becomes difficult to tell which image is which data. An effective way to solve this problem is to use annotation information that allows data to be linked directly to images. For example, Patent Document 1 discloses a method of associating real objects with information regarding the objects and recording and storing the positional relationships between the objects.

However, in the conventional technique disclosed in Patent Document 1 mentioned above, since it does not determine "what" the subject is, the units and numerical values recorded in the annotation information may not be suitable for the subject. Furthermore, the units used may differ even for the same subject depending on the user or industry.

Patent Application No. 2019-181923

The present invention has been made in view of the above problems, and it is an object of the present invention to record units suitable for a subject in annotation information.

In order to achieve the above object, the image processing device of the present invention includes a detection means for detecting a subject from an image, a measurement means for measuring width and length from an image in a common unit, and an industry field recorded by a user. a selection means for selecting, a determination means for determining a numerical value and a unit by summarizing the subject information output from the detection means, the measurement results output from the measurement means, and the industry information selected by the selection means; It has a recording means for collectively recording the numerical values and units output from the judgment means, the measurement results, and industry information in annotation information, and the standard annotation information is recorded in common units, and the industry information is recorded in common units. The annotation information is recorded in units according to the industry.

According to the present invention, the numerical value of the subject and its unit recorded in the annotation information are appropriate. In addition, when modifying annotation information, the effort of unit conversion and associated numerical value conversion can be saved.

A block diagram showing a schematic configuration of a digital camera in this embodiment Flowchart showing the entire process of recording annotation information on an image of a digital camera in this embodiment Flowchart showing a processing pattern for acquiring images and subject information of a digital camera in this embodiment A diagram showing an overview of the recorded contents of an image file recorded in an image of a digital camera in this embodiment A diagram showing a method of displaying images and annotation information of a digital camera in this embodiment Flowchart showing the process of correcting annotation information in this embodiment

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

Further, in the following description, a digital camera will be exemplified as an image output device that uses an inference model to classify a subject, but it goes without saying that the image output device of the present invention is not limited to a digital camera. The image output device of the present invention may be any device as long as it reads an image recorded in a recording device and displays it on a display device, and may be, for example, a smartphone, a tablet PC, etc. .

FIG. 1 is a block diagram showing a configuration example of a digital camera 100 according to an embodiment of the present invention.

The barrier 10 is a protective member that covers the imaging unit including the photographing lens 11 of the digital camera 100 to prevent the imaging unit from becoming dirty or damaged, and its operation is controlled by the barrier control unit 43. The photographic lens 11 forms an optical image on the imaging surface of the image sensor 13 . The shutter 12 has an aperture function. The image sensor 13 is composed of, for example, a CCD or a CMOS sensor, and converts an optical image formed on an imaging surface by the photographing lens 11 via the shutter 12 into an electrical signal.

The A/D converter 15 converts the analog image signal output from the image sensor 13 into a digital image signal. The digital image signal converted by the A/D converter 15 is written into the memory 25 as so-called RAW image data. At the same time, development parameters corresponding to each RAW image data are generated based on the information at the time of photographing, and written into the memory 25. The development parameters are comprised of various parameters used in image processing for recording, such as the JPEG format, such as exposure settings, white balance, color space, and contrast.

The timing generator 14 is controlled by the memory control section 22 and the system control section 50, and supplies clock signals and control signals to the image sensor 13, A/D converter 15, and D/A converter 21.

The image processing section 20 performs various image processing such as predetermined pixel interpolation processing, color conversion processing, correction processing, resizing processing, and image composition processing on the data from the A/D converter 15 or the data from the memory control section 22. I do. The image processing unit 20 also performs predetermined image processing and calculation processing using the image data obtained by imaging, and provides the obtained calculation results to the system control unit 50. The system control unit 50 controls the exposure control unit 40 and the focus control unit 41 based on the provided calculation results, thereby performing AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing. Realize.

The image processing unit 20 also performs predetermined calculation processing using the image data obtained by imaging, and also performs AWB (auto white balance) processing based on the obtained calculation results. Furthermore, the image processing unit 20 reads the image data stored in the memory 25 and performs compression processing or expansion processing. Here, JPEG method, MPEG-4 AVC or HEVC (High Efficiency Video Coding), reversible compression for uncompressed RAW data, etc. are performed. The image processing unit 20 then writes the processed image data into the memory 25.

The image processing unit 20 also performs predetermined arithmetic processing using image data obtained by imaging, and performs editing processing of various image data. Specifically, we use trimming processing to adjust the display range and size of an image by hiding unnecessary parts around the image data, and enlarge or reduce the size of image data and screen display elements. You can perform resizing processing to change. Furthermore, it is possible to perform RAW development, which involves reversibly compressing uncompressed RAW data, applying image processing such as color conversion to data that has undergone compression processing or decompression processing, and converting it to JPEG format to create image data. can. Further, it is possible to perform a video clipping process in which a specified frame of a video format such as MPEG-4 is clipped, converted to JPEG format, and saved.

The image processing unit 20 also includes a composition processing circuit that composes a plurality of image data. In this embodiment, in addition to simple additive synthesis or additive average synthesis, images with the brightest or darkest values are selected in each region of image data to be synthesized, and the images are synthesized to create one piece of image data. It is possible to perform comparatively bright and comparatively dark synthesis processing.

The image processing unit 20 also performs a process of superimposing an OSD (On-Screen Display) such as a menu or arbitrary characters to be displayed on the display unit 23 together with the image data for display.

Furthermore, the image processing unit 20 uses the input image data and the distance information to the subject obtained from the image sensor 13 at the time of shooting to detect the subject present in the image data, and detect the subject area. Performs subject detection processing to detect. As the information that can be detected, it is possible to obtain detection information such as a region such as a position and a size within an image, a tilt, and a probability.

The memory control section 22 controls the A/D converter 15, the timing generator 14, the image processing section 20, the image display memory 24, the D/A converter 21, and the memory 25. The RAW image data generated by the A/D converter 15 is written into the image display memory 24 or the memory 25 via the image processing section 20 and the memory control section 22, or directly via the memory control section 22.

The image data for display written in the image display memory 24 is displayed via the D/A converter 21 on the display section 23 constituted by a TFT LCD or the like. By sequentially displaying image data obtained by imaging using the display unit 23, it is possible to realize an electronic finder function that displays a live image.

The memory 25 has a storage capacity sufficient to store a predetermined number of still images and moving images for a predetermined period of time, and stores captured still images and moving images. Furthermore, the memory 25 can also be used as a work area for the system control unit 50.

The exposure control section 40 controls the shutter 12 having an aperture function. Further, the exposure control section 40 also has a flash dimming function by interlocking with the flash 44. The focus control section 41 performs focus adjustment by driving a focus lens (not shown) included in the photographing lens 11 based on instructions from the system control section 50 . The zoom control unit 42 controls zooming by driving a zoom lens (not shown) included in the photographic lens 11. The flash 44 has an AF auxiliary light projecting function and a flash dimming function.

The system control unit 50 controls the entire digital camera 100. The nonvolatile memory 51 is an electrically erasable/recordable nonvolatile memory, such as an EEPROM. Note that the nonvolatile memory 51 records not only programs but also map information and the like.

The shutter switch 61 (SW1) is turned on during the operation of the shutter button 60, and instructs the start of operations such as AF processing, AE processing, AWB processing, and EF processing. The shutter switch 62 (SW2) is turned ON upon completion of the operation of the shutter button 60, and instructs the start of a series of photographing operations including exposure processing, development processing, and recording processing. In the exposure process, a signal read from the image sensor 13 is written into the memory 25 as RAW image data via the A/D converter 15 and the memory control section 22. In the development process, the RAW image data written in the memory 25 is developed using calculations in the image processing section 20 and the memory control section 22, and is written in the memory 25 as image data. In the recording process, image data is read from the memory 25, compressed by the image processing section 20, and after storing the compressed image data in the memory 25, it is written to the external recording medium 91 via the card controller 90.

The operation unit 63 includes operation members such as various buttons and a touch panel. For example, it includes a power button, a menu button, a mode changeover switch for switching between shooting mode/playback mode/other special shooting modes, a cross key, a set button, a macro button, and a multi-screen playback page break button. For example, flash setting button, single shooting/continuous shooting/self-timer switching button, menu movement + (plus) button, menu movement - (minus) button, shooting image quality selection button, exposure compensation button, date/time setting button, etc. including.

When recording image data on the external recording medium 91, the metadata generation/analysis unit 70 generates various metadata such as the Exif (Exchangeable image file format) standard attached to the image data based on the information at the time of shooting. generate. Furthermore, when the metadata generation/analysis unit 70 reads the image data recorded on the external recording medium 91, it analyzes the metadata added to the image data. Examples of the metadata include shooting setting information at the time of shooting, image data information regarding image data, and characteristic information of a subject included in the image data. Furthermore, when recording moving image data, the metadata generation/analysis unit 70 can also generate and add metadata to each frame.

The power source 80 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The power supply control section 81 supplies power supplied from the power supply 80 to each section of the digital camera 100.

The card controller 90 sends and receives data to and from an external recording medium 91 such as a memory card. The external recording medium 91 is composed of, for example, a memory card, and records images (still images, moving images) taken by the digital camera 100.

The inference engine 73 uses the inference model recorded in the inference model recording unit 72 to perform inference on the image data input via the system control unit 50. The inference model may be input from an external device such as the external device 101 through the communication unit 71 and recorded in the inference model recording unit 72, or an inference model obtained by relearning by the learning unit 74. can. Note that management versions of the inference models are maintained in the inference model recording unit 72 or the like so that each inference model can be identified when it is updated from the outside or by relearning by the learning unit 74. That's it. The inference engine 73 also has a neural network design 73a.

In the neural network design 73a, an intermediate layer (neuron) is arranged between an input layer and an output layer. Image data is input to the input layer from the system control unit 50. Several layers of neurons are arranged as the middle layer. The number of neuron layers is determined as appropriate in design, and the number of neurons in each layer is also determined as appropriate in design. The intermediate layer is weighted based on the inference model recorded in the inference model recording unit 72. Annotation information corresponding to the image data input to the input layer is output to the output layer.

This embodiment assumes an inference model that infers the classification of what kind of subject is included in an image. Using image data of various subjects and their classification results (for example, classification of animals such as dogs and cats, classification of subjects such as people, animals, plants, buildings, etc.) as training data, an inference model generated by deep learning is developed. Make use of it.

The learning unit 74 performs relearning of the inference model upon receiving a request from the system control unit 50 or the like. The learning unit 74 has a teacher data recording unit 74a, and the teacher data recording unit 74a records information regarding the teacher data to the inference engine 73. The learning section 74 can retrain the inference engine 73 using the teacher data recorded in the teacher data recording section 74a, and can update the inference engine 73 using the inference model recording section 72.

The communication unit 71 has a communication circuit for transmitting and receiving. Specifically, the communication circuit may be wireless communication such as Wi-Fi or Bluetooth (registered trademark), or wired communication such as Ethernet or USB.

A measurement process for measuring the length and width of a subject from an image in this embodiment will be explained. The depth sensor 75 is used to measure depth information (depth information) of the subject during imaging. A depth value is associated with each pixel of the captured image as depth information. The depth value that each pixel has is information that indicates the distance from the camera's viewpoint to the subject, and the depth value may be directly expressed in physical units such as meters, or the distance from the camera's viewpoint to the subject. The distance may be normalized and expressed in a range of 0 to 1. The length per pixel is calculated based on the set resolution value. Then, with the captured image as input, the image processing section 20 performs processing such as binarization of the image to detect edges of the subject. From the extracted edges of the object, the number of pixels between the edges is counted, and the length of the object is measured based on the length per pixel calculated in advance. Note that the inference engine 73 may be used to infer the subject and perform measurement processing. Further, the principle of acquiring the depth value is not limited, and a stereo camera may be used that measures the distance to the subject using a triangulation method based on a parallax image of the subject. Alternatively, it may be a TOF (Time of Flight) camera that measures the distance to the subject based on the time it takes for the subject to be irradiated with laser light and the reflected light is detected. Of course, the distance to the subject may be measured using other methods.

Next, a process up to recording annotation information on an image of the digital camera 100 in this embodiment will be explained using FIGS. 2 to 5.

FIG. 2 is an example of an overall flowchart explaining this embodiment.

FIGS. 3A to 3D show photographing processing and internal processing of the subject information acquisition means S201 in this embodiment, and are examples of processing patterns.

FIG. 4 is an overview of the recorded contents of the image file recorded in this embodiment.

FIGS. 5A and 5B are diagrams illustrating an example of a method for displaying images and annotation information of a digital camera in this embodiment. FIG. 5A shows an image and annotation information when the subject is a Japanese-style room, and FIG. 5B shows an image and annotation information when the subject is a Western-style room.

The digital camera 100 responds to the user's pressing operation of the shutter button 60, and when the shutter switch 61 (SW1) and the shutter switch 62 (SW2) are turned on, performs photographing processing and subject information acquisition means (S201). The detailed processing flow in S201 will be explained using FIG. 3(a). Note that FIGS. 3(b), (c), and (d) are obtained by changing the processing order of FIG. 3(a), and the processing order is not limited to this.

When the user displays the menu screen on the operation unit 63 and sets industry/field information, the system control unit 50A records the industry/field information in the memory 25 (S301).

Next, the digital camera 100 responds to the user's depression of the shutter button 60. That is, when the shutter switch 61 (SW1) and the shutter switch 62 (SW2) are turned on and there is an instruction to start photographing (S302: Yes), the photographing process is started (S303).

In the photographing process (S303), the focus control section 41 and the exposure control section 40 are used to perform AF (autofocus) processing and AE (automatic exposure) processing, and then the A/D converter 15 is transferred from the image sensor 13. The image signal outputted through the memory 25 is stored in the memory 25. At this time, the depth value measured by the seismic intensity sensor 75 is also stored in the memory 25 together with the image signal. Furthermore, image data is created by compressing the image signal stored in the memory 25 in JPEG format or MPEG-4 HEVC format using compression processing included in the image processing section 20 according to the user's settings.
Next, the image processing unit 20 performs object detection processing on the image signal stored in the memory 25 to obtain detection information of the object included in the image (S304).

Next, subject inference processing is performed on the image signal stored in the memory 25 using the inference engine 73. A subject area included in the image data is specified from the image signal data stored in the memory 25 and the subject detection information acquired in S304, and a classification result of subjects included in the subject area is output (S305). At the time of inference, in addition to the inference result, information related to inference processing, such as debugging methods and logs for operations during inference, may be output.

Next, using the depth value measured in S303, the subject detection information obtained in S304, and the inference result obtained in S305 as input, the calculation unit 76 measures the height, length, width, etc. of the subject (S306). . In addition, when measuring, numerical values are calculated in common units.

Next, the digital camera 100 responds to the user's depression of the shutter button 60. That is, when the shutter switch 61 (SW1) and the shutter switch 62 (SW2) are turned on and the shooting instruction is continued (S307: No), the shooting process is started (S303). Thereafter, if the shutter switch 62 (SW2) is turned OFF and the shooting instruction is not continued (S307: Yes), the shooting process ends (S201 ends).

Next, based on the object detection information acquired in S304, the object measurement information acquired in S306, and the industry/field information acquired in S301, the determination unit 77 determines the unit to be recorded in the industry annotation information, and The numerical value of the object measurement result is converted (S202). At that time, numerical values in common units are also retained for recording standard annotations. Note that if the user does not set industry/field information in S301, only numerical values in common units are retained for recording standard annotation information.

Next, the image signal data stored in the memory 25 and the numerical values and units of the standard annotation information and industry annotation information calculated in S202 are recorded on the external recording medium 91 (S203). The image file recorded here is recorded in the format shown in FIG.

The image size file 400 recorded in this embodiment is composed of at least the following four areas. That is, an area 401 for storing metadata according to the Exif standard, an image data area 408 for recording compressed image data, standard annotation information 409 for recording annotation information, and industry annotation information 410. In the industry annotation information 410, there is always one annotation information for each industry. For example, if the user instructs the image file 400 to record in JPEG format, the image data created in S303 is recorded in the image data area 408 in JPEG format, and the Exif data 401 is recorded in the APP1 marker or the like. ing. Further, standard annotation information 409 and industry annotation information 410 are recorded in the APP1 marker and the like. Furthermore, when the user instructs recording in HEIF (High Efficiency Image File Format) format, the data is recorded in HEIF file format. The Exif data 401 and annotation information 410 are recorded in a Meta data Box or the like. Similarly, when recording in RAW format is instructed, Exif data 401, standard annotation information 409, and industry annotation information 410 are recorded in a predetermined area such as a Meta data Box.

The object detection information (obtained in S304) is recorded using the metadata generation/analysis unit 70 in a MarkerNote 405 in which manufacturer-specific metadata included in the Exif data 401 can be written in a private format in principle. In addition, if there is a subject classification result that is the inference result obtained in S305, a management version of the current inference model held in the inference model recording unit 72, debug information, etc., they are stored in the MerkerNote 405 as the inference model management information 407. Record within.

The inference result (obtained in S305) is recorded in standard annotation information 409 and industry annotation information 410. The standard annotation information 409 and the industry annotation information 410 are recorded in the positions indicated by the standard annotation information link 403 and the industry annotation information link 404 in the annotation link information storage tag 402. In this embodiment, it is assumed that the annotation information is written in a text format such as XML or JSON.

The image size file 400 recorded in this embodiment is displayed as shown in FIGS. 5(a) to 5(d). The JPEG format image data saved in the image data area 408 is displayed as an image 500 of a Japanese-style room and an image 503 of a Western-style room. The standard annotation information 409, as shown in 501 in FIG. 5(a) or 504 in FIG. 5(c), displays internally held object information and measurement information of the object in standard units (in this example, m ² ). Ru. The industry annotation information 410 stores internally held object information and measurement information of the object in industry units (in this example, a Japanese-style room is a tatami mat). , Western-style rooms are displayed in m ² ). The annotation information to be displayed is determined by the user's operation in S301. As a result, the numerical value of the subject and its unit recorded in the industry annotation information are appropriate for the industry. Note that the display formats shown in FIGS. 5(a) to 5(d) are just examples, and the display location and display method are not limited to these.

Next, a process for modifying recorded annotation information in this embodiment will be described using FIG. 6.

FIG. 6 is an example of a flowchart for a user to modify annotation information.

The digital camera 100 responds to the operation unit 63 by the user on the playback screen as shown in FIGS. The industry/field information is confirmed (S602).

Based on the object detection information corresponding to the coordinate information selected in S601 and the industry/field information acquired in S602, the object information (numeric value and unit) in the annotation information of the corresponding industry stored in the image file 400 is determined. is displayed (S603). For example, if the industry/field information acquired in S602 is standard annotation information, the subject information of the standard annotation information 409 is displayed in common units and numerical values. Further, if the industry/field information acquired in S602 is industry annotation information, the subject information of the industry annotation information 410 is displayed in industry units and numerical values.

Next, in response to the operation unit 63 by the user, if the user has completed modifying the numerical value (S604: Yes), adjust the numerical value of the other annotation information that is not currently displayed to the unit saved in the corresponding annotation information. Perform numerical conversion. For example, when the standard annotation information 409 is displayed and the numerical information of the subject A is modified by the user, the numerical information of the subject A in the industry annotation information 410 is modified to match the industry unit. Furthermore, if there is a plurality of pieces of industry annotation information, the same process is executed for all pieces of industry annotation information. This saves the effort of unit conversion and associated numerical value conversion when modifying annotation information. Note that the processing order in FIG. 6 is not limited to this.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the invention.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (9)

a detection means for detecting a subject from an image;
A measuring means for measuring width and length from an image in common units;
a selection means for selecting an industry field;
Subject information output from the detection means,
a determination means for determining a numerical value and a unit by summarizing the measurement results output from the measurement means and information regarding the industry selected by the selection means;
a recording means for collectively recording the numerical values and units output from the determination means, the measurement results, and information regarding the industry in annotation information;
A recording device that records common units and numerical values in standard annotation information, and records units and numerical values according to the industry in industry-specific annotation information. 2. The recording device according to claim 1, wherein standard annotation information and industry annotation information associated with the subject of the recorded image are displayed. 2. The recording device according to claim 1, wherein the recording device identifies a subject and displays an appropriate unit when the user records the annotation information for industry use. A claim characterized in that when a user modifies a numerical value of standard annotation information or industry annotation information, the numerical value is converted to match the unit used in the other annotation information and recorded in the other annotation information. The recording device according to item 1. A method for controlling a recording device, the method comprising:
a detection step of detecting a subject from the image;
A measurement step that measures the width and length from the image in common units,
a selection step of selecting an industry sector;
a determination step of determining a numerical value and a unit by summarizing the subject information regarding the subject detected in the detection step, the measurement result output from the measurement means, and the information regarding the industry selected by the selection means;
a recording step for collectively recording the numerical value and unit output in the judgment step, the measurement result, and information regarding the industry in annotation information;
A recording device control method that records common units and numerical values in standard annotation information, and records units and numerical values according to the industry in industry-specific annotation information. 6. The method of controlling a recording apparatus according to claim 5, further comprising displaying standard annotation information and industry annotation information linked to the subject of the recorded image. 2. The method of controlling a recording apparatus according to claim 1, further comprising identifying a subject and displaying an appropriate unit when the user records the annotation information for industry. A claim characterized in that when a user modifies a numerical value of standard annotation information or industry annotation information, the numerical value is converted to match the unit used in the other annotation information and recorded in the other annotation information. Item 1. A method for controlling a recording device according to item 1. A computer-readable program for causing a computer to function as each means of the recording device according to any one of claims 1 to 5.

Images