WO2022102140A1 - 荷電粒子ビーム装置およびレポート作成方法 - Google Patents
荷電粒子ビーム装置およびレポート作成方法 Download PDFInfo
- Publication number
- WO2022102140A1 WO2022102140A1 PCT/JP2020/042664 JP2020042664W WO2022102140A1 WO 2022102140 A1 WO2022102140 A1 WO 2022102140A1 JP 2020042664 W JP2020042664 W JP 2020042664W WO 2022102140 A1 WO2022102140 A1 WO 2022102140A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- conversation
- report
- input
- user
- image acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
- H01J37/00—Discharge tubes with provision for introducing objects or material to be exposed to the discharge, e.g. for the purpose of examination or processing thereof
- H01J37/26—Electron or ion microscopes; Electron or ion diffraction tubes
- H01J37/261—Details
- H01J37/265—Controlling the tube; circuit arrangements adapted to a particular application not otherwise provided, e.g. bright-field-dark-field illumination
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
- H01J37/00—Discharge tubes with provision for introducing objects or material to be exposed to the discharge, e.g. for the purpose of examination or processing thereof
- H01J37/02—Details
- H01J37/22—Optical, image processing or photographic arrangements associated with the tube
- H01J37/222—Image processing arrangements associated with the tube
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
- H01J2237/00—Discharge tubes exposing object to beam, e.g. for analysis treatment, etching, imaging
- H01J2237/22—Treatment of data
- H01J2237/221—Image processing
Definitions
- the present invention relates to a charged particle beam device and a report creation method.
- Patent Document 1 discloses a technique in which an icon is displayed on an observation image, and this icon provides a link to a data object including text and voice recording. Specifically, it is described that the voice recording is embedded in the observation image and layered to enable more efficient acquisition and analysis of the image data of the object.
- Patent Document 2 image data is acquired by extracting a still image from a moving image at a predetermined timing, and from the sound recorded in synchronization with the moving image, the sound is output for each speech section divided by a silent section.
- a technique is disclosed in which audio data is acquired by extraction, and image data and audio data are associated and recorded.
- the electron microscope device generally records the photographed image in association with the observation condition, but the process until the observation image is photographed needs to be manually recorded based on the experiment notebook or the like.
- the process up to shooting is not accurately recorded due to omission of entry or the like, the process becomes ambiguous and the reliability of the data is impaired. Therefore, there is a possibility that problems such as difficulty in performing a reproduction experiment may occur.
- an object of the present invention is to provide a charged particle beam device capable of easily ensuring the validity of an observation image.
- the charged particle beam device of the present invention includes an image acquisition device for acquiring a captured image of a sample and an input / output device for operating the image acquisition device.
- the input / output device generates a conversation manuscript that converts the conversation uttered by the user into character string data, and recognizes the user's conversation intention including instructions from the conversation manuscript to the image acquisition device.
- the report creation period is defined by the history holding unit that records the response of the image acquisition device to the instruction to the image acquisition device as history information and the timing when the user gives an instruction including the intention to save the captured image to the image acquisition device.
- the previous report compares the shooting conditions at the timing when the instruction including the intention to save the captured image this time is given and the shooting conditions at the timing when the instruction including the intention to save the previously shot image is given.
- Extract the common conditions and difference conditions of the shooting conditions between the creation period and the current report creation period and create a report that associates the history information divided for each report creation period, the shot images corresponding to the history information, and the difference conditions. It has a difference analysis unit that outputs information and a report holding unit that creates a report for each report creation period based on the report creation information and records the created report.
- FIG. 1 is a diagram showing a configuration example of a charged particle beam device according to the first embodiment of the present invention.
- the charged particle beam device 100 according to the present embodiment is a device that irradiates a sample with a charged particle beam.
- an apparatus for acquiring an observation image of a sample by irradiating the sample with a charged particle beam such as a scanning electron microscope (SEM)
- SEM scanning electron microscope
- this embodiment is assumed to be used by a plurality of people.
- a case where the first user 101 and the second user 102 use it will be described as an example, but it can be used by three or more people.
- the charged particle beam device 100 includes an image acquisition device 103, a first input / output device 110, a second input / output device 130, and the like.
- the first input / output device 110 is a device used by the first user 101
- the second input / output device 130 is a device used by the second user 102.
- the image acquisition device 103, the first input / output device 110, and the second input / output device 130 are connected to each other by wire, wireless, or a network (for example, the Internet or an internal network), and are described later in conversation manuscripts, conversation intentions, and control signals.
- Various information such as shot images and shooting conditions can be transmitted and received to each other.
- the first input / output device 110 and the second input / output device 130 may be referred to as input / output devices.
- the image acquisition device 103 includes a charged particle gun (for example, an electron gun) that irradiates a sample with charged particles (for example, an electron gun), a lens that controls the trajectory of the charged particles, a stage on which the sample is placed, and the image acquisition device 103 that is emitted from the sample after irradiation with the charged particles. It is equipped with a detector that detects secondary particles, etc., and a communication interface that communicates with input / output devices (both are not shown). The image acquisition device 103 irradiates charged particles from a charged particle gun based on a control signal received from the input / output device, and detects secondary particles emitted from the sample with a detector.
- a charged particle gun for example, an electron gun
- the image acquisition device 103 generates an observation image of the sample based on the secondary electrons detected by the detector.
- the image acquisition device 103 transmits the generated observation image to the input / output device.
- the image acquisition device 103 may store the generated observation image in a storage device (not shown) in the image acquisition device 103.
- the first input / output device 110 and the second input / output device 130 have the same configuration. Therefore, the configuration of the input / output device will be described by taking the first input / output device 110 as an example.
- the first input / output device 110 receives an input operation by the first user 101 and controls the image acquisition device 103 by transmitting a control signal based on the input operation.
- the first input / output device 110 receives various information such as an observation image generated by the image acquisition device 103 and state data indicating the state of the image acquisition device 103 from the image acquisition device 103.
- the first input / output device 110 shares the process of capturing the observation image and the observation image with the second input / output device 130.
- the first input / output device 110 includes a speaker 111, a microphone 112, an instruction analysis unit 113, a control unit 114, a history holding unit 115, a difference analysis unit 116, a report holding unit 119, and a monitor 120.
- the first input / output device 110 includes a microphone 112, a speaker 111, and a monitor 120, which are interfaces for a man machine.
- the first input / output device 110 is used for a computer such as a CPU that performs calculations of the instruction analysis unit 113 and the difference analysis unit 116, a storage device that realizes the functions of the history holding unit 115 and the report holding unit 119, an image acquisition device 103, and an experiment.
- a communication interface is required to send and receive information to and from the input / output devices of other participating users. Therefore, as the input / output device, for example, an information processing terminal such as a smartphone, a tablet terminal, or a personal computer (PC) can be used.
- an information processing terminal such as a smartphone, a tablet terminal, or a personal computer (PC) can be used.
- the input / output device 110 used by the first user 101 and the input / output device 130 used by the second user 102 have the same function. However, time synchronization needs to be performed between these devices.
- the microphone 112 acquires the voice instructed by the first user 101 to the charged particle beam device 100 and the voice discussed with other users including the first user 101 and the second user 102.
- the microphone 112 converts the acquired voice into voice data and outputs the voice data to the instruction analysis unit 115. In this way, information including the user's thoughts is input to the charged particle beam device 100 as voice data.
- the input device 121 is a device for inputting a document by a user's operation, such as a keyboard or a mouse.
- the input device 121 may be, for example, a software keyboard displayed on the monitor 120.
- the software keyboard may be operated by a mouse or by a monitor 120 having a touch panel function.
- the first user 101 can perform an input operation by inputting a document into the instruction input area 215 using, for example, a keyboard or the like and clicking the transmission button 216 using a mouse or the like. In this way, the user can perform an input operation to the input / output device 100 by using both the voice input via the microphone 112 and the input of text data and the like by the input device 121.
- the instruction analysis unit 113 analyzes the conversation uttered by the user input from the microphone 112, and extracts the instruction content to the image acquisition device 103 from the conversation.
- the instruction analysis unit 113 performs language understanding for the voice data input from the microphone 112.
- the instruction analysis unit 113 is a deep learning model (first deep learning) learned by using a pair of a conversation for learning (voice data) and a manuscript for learning (character string data) which is a manuscript of the conversation. Using the model), a conversation manuscript as character string data is generated from the conversation (voice data) of the first user 101.
- the deep learning model used here is, for example, long / short-term memory (LSTM: Long-Short Term Memory), but is not limited thereto.
- the instruction analysis unit 113 learns a pair of a learning manuscript (character string data) and a learning preparation diagram corresponding to the learning manuscript, which is a deep learning model (second deep learning model), for example, recurrent.
- a learning manuscript character string data
- a learning preparation diagram corresponding to the learning manuscript which is a deep learning model (second deep learning model), for example, recurrent.
- RNN recurrent neural network
- the conversation intention of the first user 101 is recognized and generated from the conversation manuscript generated by the deep learning model.
- the instruction analysis unit 113 performs language understanding for the voice data, and outputs the conversation manuscript and the conversation intention to the control unit 114 and the history holding unit 115.
- the control unit 114 controls the image acquisition device 103 based on the processing result of language understanding for the voice data in the instruction analysis unit 113.
- the control unit 114 generates a control signal of the image acquisition device 103 according to the conversation intention of the first user 101 input from the instruction analysis unit 113, and transmits the generated control signal to the image acquisition device 103.
- the control unit 114 receives a response signal from the image acquisition device 103 as a control result by the control signal.
- the control unit 114 outputs the received response signal to the speaker 111, the history holding unit 115, the difference analysis unit 116, and the report holding unit 119, respectively.
- the control unit 114 may generate a signal based on the response signal and output the generated signal.
- the content of the response signal is transmitted from the speaker 111 to the first user 101 by voice.
- the first user 101 can confirm, for example, whether or not the content instructed by the conversation is accurately transmitted to the image acquisition device 103.
- the control unit 114 may display the response of the image acquisition device 103 to the conversation of each user and the instruction of the user as text on the monitor.
- control unit 114 transmits the conversation manuscript and the conversation intention of the conversation issued by the first user 101 to the input / output devices of other users participating in the experiment. Similarly, the control unit 114 receives the conversation manuscript and the conversation intention of the conversation uttered by another user from the input / output device of the user who uttered the conversation. The control unit 114 outputs the conversation manuscript and the conversation intention of another user to the history holding unit 115.
- the control unit 114 also has a function as a communication interface for transmitting and receiving information between the first input / output device 110 and the image acquisition device 103. Further, the control unit 114 also has a function as a communication interface for transmitting / receiving information between the first input / output device 110 and the second input / output device 130.
- the communication interface function may be provided separately from the control unit 114. Note that different communication interfaces may be used between the image acquisition device 103 and the second input / output device 130.
- the control unit 114 realizes each functional block of the first input / output device 110 by executing a software program on a computer such as a CPU.
- a part of the functional block of the input / output device 110 may be configured by hardware.
- the history holding unit 115 includes a conversation manuscript and conversation intention of the first user 101 input from the instruction analysis unit 113, a response (for example, a response signal in a command format) input from the control unit 114, and a conversation manuscript and conversation of another user. Record the intention etc. as history information.
- the history holding unit 115 records the conversation manuscript, the conversation intention, and the response signal in association with the user ID of the user who originated the conversation and the time when the conversation was performed. That is, the history information includes the user ID of the user who started the conversation and the time when the conversation was performed.
- the history holding unit 115 and the report holding unit 119 are provided with a storage device.
- the storage device may be, for example, a non-volatile memory such as a flash memory, or a volatile memory such as SRAM or DRAM.
- the difference analysis unit 116 divides the report creation period with the timing when the user gives an instruction including the intention to save the captured image as a boundary.
- the history information recorded in the history holding unit 115, and the captured image 117 and the photographing condition 118 output from the control unit 114 are input to the difference analysis unit 116.
- the difference analysis unit 116 compares the shooting conditions in the current report creation period with the shooting conditions in the previous report creation period, and the common conditions of the shooting conditions between the previous report creation period and the current report creation period and Extract the difference condition.
- the difference analysis unit 116 outputs the history information divided for each report creation period, the captured image 117 corresponding to the history information, and the information in which the difference conditions and the like are associated with each other as the report creation information to the report holding unit 119.
- the common conditions of shooting conditions are separately output to the report holding unit 119.
- the captured image 117 is a plurality of images saved by the image acquisition device 103.
- the shooting condition 118 is a condition such as a control value set in the image acquisition device 103 or a measurement value obtained from the sensor when the image acquisition device 103 stores the captured image 117.
- the report holding unit 119 creates and records a report using the report creation information input from the difference analysis unit 116 and the common conditions of the shooting conditions. At that time, it is desirable that the report holding unit 119 records the report in a format that can be viewed on the monitor 120 or the like so that the report can be displayed smoothly when instructed to write the report, which will be described later. In addition, the report holding unit 119 records back data in association with the report creation information so that display / non-display can be dynamically switched for each item of the report.
- the first user 101 can view the report output and displayed on the monitor 120 of the first input / output device 110. Further, the report may be output and printed on a printer capable of printing on a paper medium. ⁇ Operation screen of input / output device>
- FIG. 2 is a diagram showing an example of an operation screen of the input / output device of FIG.
- the operation screen displayed on the first input / output device 110 will be mainly described, but if necessary, the case where the operation screen is applied to the second input / output device 130 may be described.
- the operation screen 200 is displayed on the monitor 120, and the conversation display area 201, the instruction input area 215, the transmission button 216, the observation image display area 220, the condition display area 221, the login / logout button 230, and the mute. Includes button 231, report write button 232, and report read button 234.
- the conversation display area 201 the text corresponding to the conversation (conversation manuscript) of the user of the charged particle beam device 100 and the response (response signal) from the image acquisition device 103 to the user's instruction is displayed.
- the account name 202 of the first user 101 and the input balloon 203 of the conversation of the first user 101 are associated with each other in the conversation display area 201, and one set is set. It is displayed in.
- the content of the balloon 203 is a text display of the conversation manuscript of the first user 101 output from the instruction analysis unit 113.
- the users participating in the experiment talk about instructions to the image acquisition device 103 while referring to the observation image displayed in the observation image display area 220 and the shooting conditions of the shot image displayed in the condition display area 221 described later. It is done by.
- the conversation of the balloon 203 is "align the optical axis".
- the instruction analysis unit 113 generates a conversation manuscript of the first user 101 from the voice input from the microphone 112, and outputs the generated conversation manuscript to the control unit 114.
- the control unit 114 outputs the input conversation document to the monitor 120, and the monitor displays the input conversation document on the left side of the conversation display area 201 of the operation screen 200.
- control unit 114 transmits the input conversation manuscript to the second input / output device 130.
- the control unit of the second input / output device 130 outputs the received conversation document of the first user 101 to the monitor, and the monitor displays the input conversation document in the conversation display area of the operation screen. In this way, the conversations of all users are displayed in real time on each input / output device of the participants participating in the experiment, and the content of instructions to the image acquisition device 130 can be shared.
- the image acquisition device 103 transmits a response signal not only to the first input / output device 110 but also to the input / output devices of all the users participating in the experiment. As a result, the response contents of the image acquisition device 103 to the user's instruction are displayed on the input / output devices of all the users.
- the account name 202 is a character string display of an ID that identifies the first user 101, and is displayed as "participant A" in FIG. 2, for example.
- the time when the voice is input may be displayed together with the account name 202 and the balloon 203.
- the instruction analysis unit 113 recognizes that this conversation is an instruction to the image acquisition device 103 from the conversation "aligning the optical axis" of the balloon 203, extracts the conversation intention of this conversation, and controls the control unit. Output to 114.
- the control unit 114 transmits the control signal generated based on the conversation intention input from the instruction analysis unit 113 to the image acquisition device 103.
- the balloon 204 in FIG. 2 is a textualized response from the image acquisition device 103 to the instruction "align the optical axis" from the first user 101.
- the image acquisition device 103 transmits a response signal to the control signal to the control unit 114 of the first input / output device 110.
- the control unit 114 outputs the received response signal to the monitor 120, and the monitor displays the text corresponding to the input response signal on the right side of the conversation display area 201 as a response to the user's instruction.
- the image acquisition device 103 transmits a response signal not only to the first input / output device 110 but also to the input / output devices of all the users participating in the experiment.
- the input / output devices of all users display the response contents of the image acquisition device 103 to the instructions of each user.
- the account name corresponding to the balloon 204 is displayed as "voice operation system" in FIG. 2, but if the user recognizes that the response is from the image acquisition device 103, other notations are used. But it doesn't matter.
- the response of the balloon 204 is "Align".
- the image acquisition device 103 responds to the instruction of the first user 101 to "align the optical axis” by performing “alignment” of the optical axis. After that, although not shown in FIG. 2, the image acquisition device 103 aligns the optical axis and transmits the aligned captured image to the input / output devices of all users.
- the first user 101 instructs the image acquisition device 103 to take a picture by increasing the magnification by the conversation of the balloon 205.
- the image acquisition device 103 responds by increasing the magnification to take a picture, and this response is displayed as a balloon 206.
- the image acquisition device 103 transmits the captured image captured at the changed magnification and the changed imaging conditions to the respective input / output devices.
- a report is created. In this report, conversations of balloons 203 to 206, conversation intentions corresponding to instructions by conversations of balloons 203 and 205, captured images acquired by instructions of conversation of balloon 205, shooting conditions at that time, difference conditions, and common conditions are included. Etc. are included.
- the first user 101 and the second user 102 look at the captured image displayed on the respective input / output devices, the first user 101 gives an opinion about the new captured image through the conversation of the balloon 207 to the second user 102. I have a question. Since the conversation "How is it?" In the balloon 207 does not give an instruction to the image acquisition device 103, the instruction analysis unit 113 creates only the conversation manuscript and does not create the conversation intention.
- the second user 102 states the view that "I want to clarify the internal structure a little more" in the conversation of the balloon 208.
- the instruction analysis unit of the second input / output device 130 creates only the conversation manuscript because it does not give an instruction to the image acquisition device 103.
- the conversation intention is not created.
- the instruction analysis unit of the second input / output device 130 creates and outputs a conversation manuscript and a conversation intention corresponding to the conversation of the balloon 209.
- the image acquisition device responds that the brightness is adjusted and the image is taken in the response of the balloon 210. Then, the image acquisition device 103 acquires a captured image whose brightness is adjusted, and transmits the captured image and the capture condition to the respective input / output devices.
- the conversation manuscript and the conversation intention are created based on the conversation generated by the user has been described, but the manuscript is input to the instruction input area 215 using an input device 121 (including software) such as a keyboard.
- an input device 121 including software
- the send button 216 By inputting and instructing with the send button 216, it may be assumed that the user has input by voice.
- the input manuscript input by the user becomes a conversation manuscript, and the instruction analysis unit 113 creates a conversation intention based on this input manuscript. That is, in this case, the instruction analysis unit 113 does not have to create the conversation manuscript.
- the captured image acquired by the image acquisition device 103 is displayed in real time as an observation image.
- the first user 101 and the second user 102 refer to the observation image and perform conversations between users and give instructions to the image acquisition device 103.
- condition display area 221 the imaging conditions of the captured image set in the image acquisition device 103, that is, the observation conditions of the sample are displayed.
- condition display area 221 for example, control values as shown in FIG. 2, conditions for measured values obtained from the sensor, and the like are displayed as shooting conditions for the shot image.
- the first user 101 or the second user 102 determines the condition to be set next based on these conditions, and gives an instruction to the image acquisition device by voice or document input from the input device.
- the shooting conditions include, for example, a shooting magnification, an acceleration voltage, a working distance (WD), a detection signal, a stage position (X, Y), and the like.
- the detection signal include secondary electrons (SE), backscattered electrons, X-rays, Auger electrons, analysis contrast, Z contrast, and focused ion beam.
- the login / logout button 230 is a button that opens a window for inputting a user ID. When entering the user ID, you may be required to enter the password if necessary. Further, when a plurality of users participate in the experiment, it may be possible to input cooperation information for linking the input / output devices of each user in this window. Examples of the cooperation information include a list of unique conference names managed by a certain server and IP addresses of connection destinations.
- the mute button 231 is a button for turning off the acquisition of audio signals. By providing the mute button 231, it is possible to prevent processing of an audio signal unrelated to the experiment. In this case, the button will be turned on only when the conversation is not input. Further, the mute button 231 may be changed to a button for starting the input of the conversation. In this case, this button will be turned on only when inputting a conversation.
- the input of conversation may be switched on / off with a keyword.
- a mute button 231 or a button for starting conversation input may be provided.
- the report write button 232 is a button for outputting the report recorded in the report holding unit 119.
- the report export button 232 When the report export button 232 is pressed, the report is displayed in a format that can be viewed on the monitor 120. The report may also be printed from the printer.
- the report read button 234 is a button used for conducting an experiment using a report that has already been generated.
- the report read button 234 is pressed, the report recorded in the report holding unit 119 is read. Further, when reading a report, first, the list of reports may be displayed on the monitor 120, and only the report selected by the user may be read from the list.
- the control unit 114 extracts the newest shooting conditions and collectively sets the extracted shooting conditions in the image acquisition device 103. Further, the difference condition may be appropriately selected from a plurality of conditions included in the report. As a result, even if the experiment is interrupted once, the experiment can be restarted under the same conditions as long as the report at that time is retained.
- the report can be read by pressing the report read button 234, and the read report can be referred to under the same shooting conditions. Re-observation is possible. As a result, if a similar observation image is obtained, the validity of the data can be proved. Alternatively, it is possible to acquire the missing data that causes the failure to determine whether the data is valid or not later. ⁇ Example of created report>
- FIG. 3 is a diagram showing an example of a report corresponding to FIG. 2.
- the report REP of FIG. 3 includes a title description area 301, a comment insertion button 302, a save button 303, a common condition description area 310, a first data set description area 320, a second data set description area 330, and the like.
- the title of the experiment input by the user is displayed.
- the user can add information that identifies the experiment, such as the date and time of the experiment, as necessary. This makes it possible to improve the readability and searchability of the experimental report.
- the comment insertion button 302 is a button used when inserting text or voice data into the report REP for the purpose of adding items missing during the experiment, correcting errors, and the like.
- the save button 303 is a button used when the output report is edited and the edited report is saved in the report holding unit 119. As a result, even when the data is organized after the experiment, the report holding unit 119 can centrally manage the reports.
- the common condition description area 310 is an area for displaying common conditions among shooting conditions.
- each item included in the common condition of the report is displayed.
- an item to be displayed as a common condition may be selected for each experiment.
- the irradiation current may be displayed as a common condition, unlike a sample for which EDS analysis is not performed.
- the first data set description area 320 displays the first captured image 321. Further, as information for explaining the difference between the first captured image 321 and the other captured images, the first captured image 321 is extracted by the difference analysis unit 116 in the vicinity of the first captured image 321 (immediately below the captured image in FIG. 3). 1 Difference condition 322 is displayed.
- the first delete button 323 is displayed on the first captured image 321. When the first delete button 323 is pressed, the first data set description area 320 is deleted from the report. At that time, a part of the display contents may be integrated with the second data set description area 330.
- the first data set description area 320 displays the first shooting process 324 showing a list of intentions, circumstances and procedures for acquiring the first shot image 321.
- the history divided by the difference analysis unit 116 as the timing boundary in which the shot image 321 is stored is shown.
- the first shooting process 324 for example, as shown in FIG. 3, the time when the conversation is started (recorded), the user ID of the user who started the conversation, the conversation manuscript, the operation of the image acquisition device 103 by the conversation, and the like are performed. included.
- the second data set description area 330 displays the second captured image 331, the second difference condition 332, and the second imaging process 334, similarly to the first data set description area 320.
- the second delete button 333 is displayed on the second captured image 331.
- the first delete button 323 When the first delete button 323 is pressed, the first captured image 321 and the first difference condition 322 are deleted from the report REP. Since the first photographing process 324 includes a process until the second photographing image 331 is obtained, it may be integrated into the second photographing process 334.
- "contrast: +1" displayed in the second difference condition 332 is an extraction of the shooting conditions changed between the first shot image 321 and the second shot image 331. Therefore, when the first data set description area 320 is erased, it is not extracted as the changed shooting condition, so that the second difference condition 332 may be left blank. ⁇ Search for the process until shooting>
- FIG. 4 is a diagram illustrating a method of searching the process up to shooting.
- the search screen 400 illustrated in FIG. 4 includes a search content input unit 401, a search button 402, a shooting process 403, a highlight 404, a shot image 405, a selection button 406, an OK button 407, and a Cancel button 408.
- Participant A participates in an experiment for observing Sample A, Sample B, and Sample C, but knows how to set the imaging conditions for Sample D to be observed for the first time. Since there is no such thing, it is assumed that the shooting conditions are set using the function to search the process up to shooting.
- the participant inputs to the user search content input unit 401, "For sample D, observe the internal structure of the plating process in the cross section cut out by milling.”, And the search button 402. Press.
- the control unit 114 determines the purpose of the observation input here.
- a search keyword the report recorded in the report holding unit 119 is searched, and sentences similar to the search keyword are extracted.
- a shooting process 403 including a conversation manuscript that matches or is similar to the search keyword is displayed as a search result.
- a portion that matches or resembles the search keyword is displayed as a highlight 404.
- the highlight 404 is shown by hatching, but the highlight 404 can also be realized by making the highlight 404 stand out by color instead of hatching.
- the captured image 405 corresponding to the photographing process 403 extracted by the search is displayed side by side with the photographing process 403.
- Figure 4 lists the datasets of the three reports extracted by the search.
- the participant A selects a data set determined to be appropriate for the sample D to be observed for the first time with the selection button 406, and presses the OK button 407.
- the control unit 114 acquires the common conditions and the difference conditions of the selected data set from the report holding unit 119, and sets these acquired conditions in the image acquisition device 103.
- the Cancel button 408 can be pressed to start observing the sample D without changing the conditions of the image acquisition device 103. Is.
- the search results in FIG. 4 show examples in which the shooting conditions are set satisfactorily in order to capture the internal structure.
- Participant A judged that the condition of the second stage from the top was appropriate by grasping the structure inside the hole, which is the same metal as the plating process, and selected the data set of the second stage from the top, and the hole. Observation is started under the condition that the inside can be acquired like the corresponding captured image.
- the instruction analysis unit 113 generates a conversation manuscript in which the conversation uttered by the user is converted into character string data, and recognizes the user's conversation intention including the instruction to the image acquisition device from the conversation manuscript. Then, the history holding unit 115 records the conversation manuscript, the conversation intention, and the response from the image acquisition device 103 to the instruction to the image acquisition device 103 as history information. Then, the difference analysis unit 116 divides the report creation period with the timing at which the user gives an instruction including the intention to save the captured image to the image acquisition device 103 as a boundary, and includes the intention to save the captured image this time.
- the shooting conditions at the timing of the instruction are compared with the shooting conditions at the timing of the instruction including the intention to save the previous shot image, and the shooting conditions between the previous report creation period and the current report creation period are compared. Extract the common condition and the difference condition of. Then, the difference analysis unit 116 outputs the history information divided for each report creation period, the captured image corresponding to the history information, and the report creation information associated with the difference condition. Then, the report holding unit 119 creates a report for each report creation period based on the report creation information, and records the created report.
- each report includes conversation content, captured images, imaging conditions, etc.
- the experiment can be easily reproduced by referring to this report, and the validity of the observed image can be easily ensured.
- the validity of the observation image is questioned, the operation up to the shooting and the conversation up to that point are recorded, so it is possible to perform a reproduction experiment as well as a clear explanation, and observation. The validity of the image can be easily shown.
- the instruction analysis unit 113 is issued by the user using the first deep learning model learned by using the pair of the learning conversation and the learning manuscript which is the manuscript of the learning conversation. Generate a conversation manuscript from the conversation you have made. According to this configuration, it is possible to understand the conversation made by the user using the deep learning model.
- the first deep learning model is long / short-term memory. According to this configuration, the existing technique can be suitably applied to the charged particle beam apparatus of the present embodiment.
- the instruction analysis unit 113 uses the second deep learning model in which the pair of the learning manuscript and the learning preparation diagram corresponding to the learning manuscript is learned to be the first deep layer. Recognize the user's conversation intention from the conversation manuscript generated by the learning model. According to this configuration, it is possible to recognize the conversation intention included in the conversation made by the user by using the deep learning model.
- the second deep learning model is a recurrent neural network. According to this configuration, the existing technique can be suitably applied to the charged particle beam apparatus of the present embodiment.
- a plurality of input / output devices are provided, a plurality of users use different input / output devices, and each user's input / output device is used by all users who participate in the experiment. The conversation you made is displayed. According to this configuration, a plurality of users can share the experiment contents even if they are in a remote place. In addition, the experiment can be carried out efficiently.
- control unit 114 reads the report recorded in the report holding unit 119, and sets the shooting conditions corresponding to the read report in the image acquisition device 103. According to this configuration, it is possible to easily perform a reproduction experiment. (Embodiment 2)
- FIG. 5 is a diagram showing a configuration example of the charged particle beam device according to the second embodiment of the present invention.
- one input / output device is used by a plurality of users.
- the input / output device 510 of FIG. 5 has a configuration in which a speaker recognition unit 501 is added to the input / output device 110 of FIG.
- the speaker recognition unit 501 is connected to the microphone 112 and recognizes a speaker who has input a conversation from the microphone 112 to the input / output device 510.
- the speaker recognition unit 501 recognizes (identifies, authenticates) the speaker who has made a conversation by speaker recognition, and outputs the recognized speaker account information 502 to the history holding unit 115.
- the speaker recognition unit 501 When performing speaker recognition, the speaker recognition unit 501 reads the learning information necessary for speaker recognition when logging in with the login / logout button 230 (FIG. 2). The speaker recognition unit 501 uses the read information to learn in advance the features included in the user's voice. Alternatively, the speaker recognition unit 501 may use a deep learning model in which the audio signal input from the microphone 112 is learned in advance.
- the speaker recognition unit 501 refers to the correspondence information that associates the user (speaker) with the account information 502, and acquires the account information 502 of the user specified by the speaker recognition. Then, the speaker recognition unit 501 outputs the acquired account information 502 to the history holding unit 115. In this case, it is possible to identify the speaker even when one microphone is shared by a plurality of users.
- the history holding unit 115 records the conversation manuscript and the conversation intention input from the instruction analysis unit 113 in association with the account information 502 input from the speaker recognition unit 501.
- Correspondence information for associating the user (speaker) with the account information 502 is recorded in the storage device in the input / output device 501.
- a microphone for each user may be provided.
- one microphone 112 is assigned to one user ID.
- device information that identifies the microphone 112 for each user is specified.
- the speaker recognition unit 501 can recognize the speaker by detecting the microphone into which the conversation is input. Further, the speaker recognition unit 501 may perform speaker recognition for the input from each microphone. This makes it possible to accurately identify the speaker even when the conversation is input from a microphone different from the one registered in advance by the user.
- one input / output device 110 is used by a plurality of users.
- the input / output device 110 includes a speaker recognition unit 501 that recognizes the user who has made a conversation. According to this configuration, it is possible to reduce the space and cost required for the input / output device by sharing the necessary hardware.
- FIG. 6 is a diagram showing a configuration example of the charged particle beam device according to the third embodiment of the present invention.
- the input / output device 610 of FIG. 6 has a configuration in which a routine extraction unit 601 is added to the input / output device 110 of FIG.
- the routine extraction unit 601 reads a plurality of reports from the report holding unit 119, and extracts operations commonly performed in a plurality of experiments from the plurality of read reports.
- routine extraction unit 601 When the routine extraction unit 601 extracts an operation that is commonly performed from a plurality of reports, the routine keyword 602 is assigned to the extracted operation and registered in the instruction analysis unit 113. Further, the routine extraction unit 601 generates a command included in the operation corresponding to the routine keyword 602 as a routine command 603. Further, the routine extraction unit 601 outputs the routine keyword 602 and the routine command 603 to the control unit 114. The control unit 114 associates the input routine keyword 602 with the corresponding routine command 603 and registers the input routine keyword 602 in the storage device as a set.
- routine commands 603 corresponding to one routine keyword is not limited to one.
- a command set consisting of multiple commands may correspond to one routine keyword.
- the instruction analysis unit 113 When the registered routine keyword 602 is input by the user, the instruction analysis unit 113 outputs the input routine keyword 602 to the control unit 114, and instructs the control unit 114 to execute the routine command 603.
- the control unit 114 uses the routine keyword 602 input from the instruction analysis unit 113 to search for the routine command 603 corresponding to the routine keyword 602.
- the control unit 114 executes a routine command that matches the search condition, and controls the image acquisition device 103.
- the routine extraction unit 601 extracts, for example, the routine keyword 602 "usual" from a plurality of reports, and for the extracted routine keyword 602, at least each command of automatic optical axis adjustment and automatic focus adjustment. Generate a command set containing the routine command 603.
- the image acquisition device 103 executes automatic optical axis adjustment and automatic focus adjustment by inputting "usual" by voice before the start of the experiment.
- the routine extraction unit 601 when the routine extraction unit 601 extracts an operation commonly performed from a plurality of reports, the routine keyword 602 is assigned to the extracted operation, and the routine keyword 602 is registered in the instruction analysis unit 113. .. Then, the routine extraction unit 601 generates a command included in the extracted operation as a routine command 603. According to this configuration, by extracting and executing the work performed by the user in the routine, it is possible to prevent the occurrence of human error during the experiment and reduce the complexity of the work. (Embodiment 4)
- FIG. 7 is a diagram showing a configuration example of the charged particle beam device according to the fourth embodiment of the present invention.
- the input / output device 710 of FIG. 7 has a configuration in which a camera 701 and an emotion analysis unit 702 are added to the input / output device 110 of FIG. 1, and a history holding unit 115 and a difference analysis unit are deleted.
- the camera 701 acquires a facial expression image 705 including the facial expression of the first user 101, and outputs the acquired facial expression image 705 to the emotion detection unit 702.
- the output of the microphone 112 is input to the emotion detection unit 702.
- the emotion detection unit 702 uses a deep learning model whose features have been learned in advance, and is based on at least one of the facial expression image 705 input from the camera 701 and the voice information of the first user 101 input via the microphone 112. , Detects the emotions of the first user 101.
- the emotion detection unit 702 detects the user's emotion from, for example, the facial expression image 705 or the conversation made by the user.
- the emotion detection unit 702 detects emotions such as "joy” and “surprise” of the user
- the emotion detection unit 702 outputs a signal corresponding to each emotion to the instruction analysis unit 113 and the control unit 114.
- the emotion detection unit 702 outputs the voice 704 to the instruction analysis unit 113 and outputs the shooting instruction 703 to the control unit 114. Further, the emotion detection unit 702 outputs the input facial expression image 705 to the report holding unit 119.
- the instruction analysis unit 113 extracts the user's impression 706 from the input voice 704, and outputs the extracted impression 706 to the report holding unit 119. Further, the instruction analysis unit 113 may generate a conversation manuscript and output it to the report holding unit 119, as in the previous embodiments.
- the report holding unit 119 creates a report associating the facial expression image 705 input from the emotion detection unit 702, the impression 706 input from the instruction analysis unit 113, the captured image 117 input from the control unit 114, and the photographing condition 118. ..
- the report is recorded in the report holding unit 119 in a viewable format.
- the report of this embodiment includes, for example, an observation image that was observed when the user was impressed, and the state (facial expression and impression) of the user at that time.
- the observation of the sample using the charged particle beam device has become the theme in the child's independent study.
- the report at this time shall require experimental results and impressions. Assuming that the reader of the report is a school teacher or a guardian, it is considered that the child is interested in what kind of observation image he / she saw and what kind of impression he / she had, or what the child looked like at the time of the experiment. Be done.
- the report of this embodiment can be used as a material for communicating the results of the child's independent study to the teacher and parents, or as a pre-material for the report prepared by the child.
- the present invention is not limited to the above embodiment and can be variously modified without departing from the gist thereof.
- the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations.
- it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. ..
Landscapes
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Debugging And Monitoring (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2022561254A JP7416975B2 (ja) | 2020-11-16 | 2020-11-16 | 荷電粒子ビーム装置およびレポート作成方法 |
| PCT/JP2020/042664 WO2022102140A1 (ja) | 2020-11-16 | 2020-11-16 | 荷電粒子ビーム装置およびレポート作成方法 |
| US18/035,589 US12525424B2 (en) | 2020-11-16 | 2020-11-16 | Charged particle beam apparatus and report creation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2020/042664 WO2022102140A1 (ja) | 2020-11-16 | 2020-11-16 | 荷電粒子ビーム装置およびレポート作成方法 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022102140A1 true WO2022102140A1 (ja) | 2022-05-19 |
Family
ID=81602150
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2020/042664 Ceased WO2022102140A1 (ja) | 2020-11-16 | 2020-11-16 | 荷電粒子ビーム装置およびレポート作成方法 |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US12525424B2 (cg-RX-API-DMAC7.html) |
| JP (1) | JP7416975B2 (cg-RX-API-DMAC7.html) |
| WO (1) | WO2022102140A1 (cg-RX-API-DMAC7.html) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20250062365A (ko) * | 2023-10-30 | 2025-05-08 | 엘지전자 주식회사 | 화자 인식 장치 및 그의 동작 방법 |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS60264034A (ja) * | 1984-06-13 | 1985-12-27 | Hitachi Ltd | 理化学装置 |
| JPH0744908A (ja) * | 1993-07-30 | 1995-02-14 | Canon Inc | 情報記録再生装置 |
| JPH0785510A (ja) * | 1993-09-13 | 1995-03-31 | Canon Inc | 情報記録再生装置 |
| JP2001084945A (ja) * | 1999-09-20 | 2001-03-30 | Hitachi Ltd | 走査形電子顕微鏡あるいは類似装置における音声入力(操作)装置 |
| US20140067102A1 (en) * | 2005-07-14 | 2014-03-06 | Zaxcom, Inc. | Virtual wireless multitrack recording system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5408852B2 (ja) * | 2007-08-09 | 2014-02-05 | 株式会社日立ハイテクノロジーズ | パターン測定装置 |
| JP5740723B2 (ja) | 2009-05-13 | 2015-06-24 | 株式会社大一商会 | 遊技機 |
| EP2450936B1 (en) | 2010-11-03 | 2013-03-13 | Carl Zeiss NTS Ltd. | Microscope system, method for operating a charged-particle microscope |
| JP6276570B2 (ja) | 2013-12-05 | 2018-02-07 | キヤノン株式会社 | 画像音声再生システム及び画像音声再生方法とプログラム |
| US12100131B2 (en) * | 2020-10-26 | 2024-09-24 | University Of Florida Research Foundation, Incorporated | Knowledge-based object localization in images for hardware assurance |
-
2020
- 2020-11-16 US US18/035,589 patent/US12525424B2/en active Active
- 2020-11-16 WO PCT/JP2020/042664 patent/WO2022102140A1/ja not_active Ceased
- 2020-11-16 JP JP2022561254A patent/JP7416975B2/ja active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS60264034A (ja) * | 1984-06-13 | 1985-12-27 | Hitachi Ltd | 理化学装置 |
| JPH0744908A (ja) * | 1993-07-30 | 1995-02-14 | Canon Inc | 情報記録再生装置 |
| JPH0785510A (ja) * | 1993-09-13 | 1995-03-31 | Canon Inc | 情報記録再生装置 |
| JP2001084945A (ja) * | 1999-09-20 | 2001-03-30 | Hitachi Ltd | 走査形電子顕微鏡あるいは類似装置における音声入力(操作)装置 |
| US20140067102A1 (en) * | 2005-07-14 | 2014-03-06 | Zaxcom, Inc. | Virtual wireless multitrack recording system |
Also Published As
| Publication number | Publication date |
|---|---|
| US12525424B2 (en) | 2026-01-13 |
| JPWO2022102140A1 (cg-RX-API-DMAC7.html) | 2022-05-19 |
| US20230420214A1 (en) | 2023-12-28 |
| JP7416975B2 (ja) | 2024-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116368785B (zh) | 智能查询缓冲机制 | |
| US11281707B2 (en) | System, summarization apparatus, summarization system, and method of controlling summarization apparatus, for acquiring summary information | |
| US11611600B1 (en) | Streaming data processing for hybrid online meetings | |
| JP2006085440A (ja) | 情報処理システム及び情報処理方法、並びにコンピュータ・プログラム | |
| KR102263154B1 (ko) | 스마트 미러 기반 얼굴 감성 표현 시스템 및 그 운용방법 | |
| JP2015104078A (ja) | 撮像装置、撮像システム、サーバ、撮像方法、及び撮像プログラム | |
| JP7096626B2 (ja) | 情報抽出装置 | |
| CN112307877A (zh) | 信息处理方法、信息处理装置及计算机可读取的存储介质 | |
| JP7416975B2 (ja) | 荷電粒子ビーム装置およびレポート作成方法 | |
| JP2021184601A (ja) | 情報処理装置 | |
| JP6091690B1 (ja) | 議会運営支援システム及び議会運営支援方法 | |
| CN112738499A (zh) | 基于ar的信息显示方法、装置、ar设备、电子设备及介质 | |
| CN120581010A (zh) | 会议纪要记录方法、装置、设备及可读存储介质 | |
| CN120455186A (zh) | 纪要生成方法、装置、智能眼镜、服务器和系统 | |
| Chauhan et al. | Establishing communication through lip reading with the aid of machine learning | |
| CN119444094A (zh) | 基于大语言模型的会议纪要生成方法、设备、介质和产品 | |
| JP7697455B2 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
| CN117456035A (zh) | 一种图像再现方法及装置 | |
| CN118098224A (zh) | 屏幕共享控制方法、装置、设备、介质以及程序产品 | |
| JPWO2022102140A5 (cg-RX-API-DMAC7.html) | ||
| CN120186144B (zh) | 一种基于人工智能的文件传输方法和系统 | |
| WO2021171613A1 (ja) | サーバ装置、会議支援システム、会議支援方法及びプログラム | |
| Liu et al. | Integrating Natural Language Processing & Computer Vision into an Interactive Learning Platform | |
| JP7627648B2 (ja) | 対話管理装置、対話管理システム及び対話管理方法 | |
| JP7691149B2 (ja) | 動画像分析システム |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20961663 Country of ref document: EP Kind code of ref document: A1 |
|
| ENP | Entry into the national phase |
Ref document number: 2022561254 Country of ref document: JP Kind code of ref document: A |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 18035589 Country of ref document: US |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 20961663 Country of ref document: EP Kind code of ref document: A1 |
|
| WWG | Wipo information: grant in national office |
Ref document number: 18035589 Country of ref document: US |