Getting Started

FEI/Thermo Fisher Scientific (TFS) electron microscopes store instrument data using Data Services software on the microscope PC (MPC). This data includes event logging (Health Monitor), system configuration, alarms (error codes) etc. The Data Services backend uses Microsoft SQL Server (and PostgreSQL on new systems) with several databases.

Various applications like Health Monitor, FEI Viewer, UEC viewer, and D2i Data Collector access this data. Since the Data Services API is proprietary and TFS does not provide SQL server credentials, data can only be accessed through Health Monitor (HM). The HM client is installed on MPC and optionally on support PCs, allowing connection to Data Services to view and export data in XML or CSV formats.

The EMHealth package provides functionality to:

  • Parse and import XML data into a TimescaleDB database

  • Visualize and analyze data using Grafana

Typical Setup

  1. Windows PC (microscope or support) with:

  • Health Monitor client

  • Scheduled task for continuous data export to a location shared with Linux PC

Tip

A single Health Monitor client (e.g. on a support PC) can connect to different microscopes if they are all on the same network.

  1. Linux PC running EMHealth with:

  • Access to the shared directory with exported files

  • Watchdog service monitoring modified XML files

  • Automatic data import pipeline

Prerequisites

Supported systems:

  • TEM Glacios

  • TEM Titan Krios G1, G2, G3i, G4

  • TEM Tundra

  • SDB Scios

  • SDB Aquilos 2+

  • pFIB Arctis

Requirements for EMHealth package:

The rest is managed by Docker/Podman and Conda environment. It’s recommended to manage Docker as a non-root user, see details.

Installation

  1. Set up Python environment and install package:

    conda create -y -n emhealth python=3.12
conda activate emhealth
git clone https://github.com/azazellochg/em_health
cd em_health
pip install -e .

  2. Edit the configuration variables in docker/.env. Make sure BACKUP_DIR exists and has 777 permissions, since the containers need to write into this host folder. Detailed information can be found here.

  3. Launch containers:

    docker compose -f docker/compose.yaml up -d

    or if you are running Podman

    podman-compose -f docker/compose.yaml up -d

Data Import

Historical Data Import

  1. [Windows] Export XML data from Health Monitor (via GUI or CLI). Be aware, an instrument can have several associated DataSources (for HM, APM, AutoCTF, AutoStar, ToolReadiness, Velox etc). You need to select one that has Software->Server parameter.

  1. Choose a date range, e.g. 1 month.

  2. Select ALL parameters.

  3. Format: XML

  4. Press Save.

_images/HM_export.png

Note

If you select a very large date range, the export may fail. Also, new systems like Krios G4 have thousands of parameters, you should split export to several shorter time ranges.

  1. Transfer file.xml to Linux and compress it using GZIP (gzip file.xml). This reduces the file size >10 times.

  2. Configure instruments in instruments.json. See help for details

  3. Import data (this may take a few minutes depending on the number of parameters and amount of data):

    emhealth import -i /path/to/file.xml.gz -s em_health/instruments.json

  4. If necessary, repeat export and import steps for other instruments.

Automated Import Setup

  1. Generate Windows batch file for all instruments in the instruments.json file:

    emhealth create-task -s em_health/instruments.json

  2. Open export_hm_data.cmd and change OUTDIR value to a full path pointing to a shared location, available from Linux PC. Make sure the file name terminates with *_data.xml

  3. [Windows] Create a new task in Task Scheduler to trigger the generated script every 30 min indefinitely. The script will keep overwriting the output xml file. See advanced setup for details

Note

The task will run only when a user is logged on. This is because in Windows the network drives are mounted on a per-user basis.

  1. Start the watchdog service, which checks the directory every 5 minutes for modified files matching *_data.xml or *_data.xml.gz:

    emhealth watch -i /path/to/xml/dir -s em_health/instruments.json -t 300

Post-Import Steps

  1. Calculate initial historical statistics for the dashboards (run this step only once!):

    emhealth db -d tem create-stats

  2. Access Grafana dashboards at http://localhost:3000

    • Login with admin account

    • Navigate to “TEM” folder for instrument dashboards