Frequently Asked Questions

EnviStor is a smart data infrastructure that integrates storage, computation, metadata, and visualization to streamline research data discovery, processing, and sharing for environmental datasets of the South Florida Region. Examples of datasets are buoy data, climate (rainfall, temperature) data, elevation, storm surge, aerial imageries, autonomous underwater vehicle captured datasets, as well as Wall of Wind video files, etc. The NSF-funded EnviStor project is a collaborative effort among the FIU Institute of Environment, Knight Foundation School of Computing and Information Sciences, FIU CED-GIS Center, Division of IT (DoIT), Sea Level Rise Solution Center, and Institute of Extreme Events. See the full list of collaborators and leads on the Home page.

Dataverse is an open-source web application designed to store, share, cite, and publish research data. It provides a platform where researchers can manage and share datasets in a structured, secure, and accessible way. Dataverse is often used by academic institutions, research organizations, and data repositories to promote transparency, reproducibility, and collaboration in research.

Key features of Dataverse:

• Dataset versioning and file-level access control
• Integration with DOIs (Digital Object Identifiers) for data citation
• Metadata support for discovery and reuse
• Data curation tools and analytics integration
• Customizable metadata schemas for different disciplines

FIU Dataverse serves as a data catalog for discovery of datasets, supporting documents, scripts, codes, tools and models in EnviStor. We aim to manage two different types of datasets — FIU researcher generated, collected, processed, or value-added data, and data downloaded from government agencies or other third parties. Every item in the FIU EnviStor should contain high-quality metadata. Including high-quality metadata while publishing/adding items to the Dataverse is crucial in ensuring the smooth transfer of these items to the Pelican, and ultimately, in making discovery of these datasets possible by the end users within the federated computing network.

For more information about viewing and editing metadata in Dataverse manually: User Guide — Dataverse.org.

The HITL (human-in-the-loop) multi-agent agentic-AI platform is part of a broader initiative by Florida International University (FIU) to enable scalable, automated, and intelligent research data management. Like many institutions worldwide, FIU serves a diverse research community with growing needs to store, process, compute, discover, and share domain-specific data.

On this platform, multiple specialized AI agents perform the repeatable work of data management — completeness and integrity checks, format and unit harmonization, standards-aligned metadata generation, QA/QC preparation, and publication packaging — while human reviewers approve scope, quality, and release. Every processing step is logged with provenance, so any published dataset can be traced back to its inputs and the decisions (human and AI) that shaped it.

The platform has been validated end-to-end on production datasets — for example, a real-time air-quality and atmospheric-chemistry archive (PurpleAir, Tempest, and NASA Pandora) is checked, cleaned, documented, and published through the full pipeline to Dataverse, ArcGIS Online, and Pelican. A companion coastal buoy pipeline has been validated end-to-end on test data and is moving toward production as part of an upcoming collaboration with the NSF CREST Center for Aquatic Chemistry & Environment, the source data owner.

See current, live-verified publication numbers for every project on the Data page.

The Open Science Pool (OSPool) is a distributed high-throughput computing (dHTC) platform operated under the Open Science Grid (OSG). It federates compute resources from multiple institutions and provides a scalable, flexible, and cost-effective environment to run large numbers of computational tasks in parallel.

• Massive scale: ideal for workloads that consist of many independent jobs (e.g., data preprocessing, simulations, model evaluations).
• Data-aware scheduling: paired with OSDF (Open Science Data Federation), OSPool can access data in-place from federated storage without unnecessary data movement.
• Federated access: supports workflows that span multiple campuses and HPC sites.

In the EnviStor framework, OSPool is leveraged to perform large-scale computation on federated datasets, especially from Pelican-based storage systems. When data is stored and registered through OSDF/Pelican, users can run jobs on it without needing to replicate the datasets across systems. This dramatically increases the efficiency of processing large environmental datasets like buoy time series, sensor data, or remote sensing outputs.

Raptor (Reconfigurable Advanced Platform for Transdisciplinary Open Research) is FIU's institutional high-performance computing (HPC) cluster, built around NVIDIA A-series GPUs and open to FIU researchers. It is designed to support data-intensive and compute-heavy research by offering configurable compute environments for diverse workflows — GPU model training, large simulations, and traditional HPC/batch jobs.

The AI Pavilion is FIU's strongest local AI/ML compute environment (hosted by the Knight Foundation School of Computing and Information Sciences), providing JupyterHub-based interactive development, model training, and inference close to EnviStor storage.

dPanther provides traditional web and database hosting for applications, dashboards, and APIs built on EnviStor data.

Data lives once in NexusOS storage; each resource reaches it in place — through a mount, S3/Ceph keys, or Pelican federation — rather than by copying data around.

Pelican is a data federation platform developed by the Partnership to Advance Throughput Computing (PATh) to support the Open Science Data Federation (OSDF). It allows institutions to expose datasets stored in POSIX filesystems, S3 buckets, or HTTP servers as part of a distributed, federated storage network.

We selected Pelican for the following reasons:

1. Federated Data Sharing — Pelican enables distributed data access without centralizing storage. Researchers can publish datasets from their local storage systems while making them accessible through a global, persistent pelican:// URL.
2. Compute-Aware Federation — Datasets registered with Pelican can be directly accessed by compute platforms like OSPool, enabling data-local computation. This reduces data movement overhead and improves performance for large-scale workflows.
3. Scalability and Flexibility — Pelican supports a wide range of storage backends (POSIX, S3, HTTP), making it compatible with FIU's existing infrastructure (e.g., Ceph, OSNexus).
4. FAIR Support via Metadata Linking — While Pelican itself has minimal metadata capabilities, it integrates seamlessly with Dataverse, which manages rich metadata. We use Dataverse for metadata discoverability and DOI assignment, and link it to data stored in Pelican, ensuring compliance with FAIR (Findable, Accessible, Interoperable, Reusable) principles.
5. Open-Source and Community-Aligned — Pelican is part of an actively maintained, open-source ecosystem that aligns with EnviStor's mission to support inter-institutional research and shared infrastructure.

When publishing datasets to FIU EnviStor, we recommend but do not require following the naming convention outlined in the table below. This file naming structure captures file type, keywords, geo name, and year of the datasets. For datasets downloaded from agencies, you may keep the original file naming conventions and structure.

FIU EnviStor accommodates various data formats:

1. Shapefiles/feature classes (zip, or .shp)
2. Spreadsheets (.csv)
3. Jupyter notebooks, Py scripts, codes (_R, _Py)
4. Imagery or tile-based raster (e.g. elevation)
5. Other (PDF, API, etc.)

The most used environmental and climate data formats and tools are as follows:

• NetCDF (.nc, .nc4) — Python: xarray, netCDF4, cfgrib; R: ncdf4, raster, stars; MATLAB: ncread, ncinfo; command-line: ncdump, NCO toolkit, cdo; GUI: Panoply, QGIS.
• HDF/HDF5 (.hdf, .h5) — Python: h5py, netCDF4, xarray; R: rhdf5, hdf5r; MATLAB: h5read; command-line: h5dump, h5ls; NASA HEG Tool for MODIS HDF.
• GeoTIFF (.tif, .tiff) — Python: rasterio, rioxarray, GDAL; R: raster, terra; GIS: QGIS, ArcGIS, GRASS GIS; command-line: gdalinfo, gdal_translate, gdalwarp.
• GRIB (.grib, .grb, .grb2) — Python: cfgrib, pygrib; command-line: wgrib2, cdo, ncl; GUI: Panoply, QGIS.
• CSV (.csv) — Python: pandas; R: readr; Excel/LibreOffice; import as points in QGIS/ArcGIS if lat/lon columns exist.
• Shapefile (.shp, .shx, .dbf) — Python: geopandas, fiona; R: sf, rgdal, sp; GIS: QGIS, ArcGIS, GRASS GIS; command-line: ogrinfo, ogr2ogr.
• GeoJSON (.geojson, .json) — Python: geopandas, folium; R: sf, geojsonio, leaflet; web: Mapbox, OpenLayers; GIS: QGIS, ArcGIS.
• Zarr — Python: zarr, xarray (native support), Pangeo stack (Dask, Xarray, Zarr); cloud-native object stores (e.g., S3); works well in JupyterHub/Dask setups for scalable analysis.

That's correct. The agentic-AI platform will be handling the processing of the data and generation of the metadata. However, the spreadsheet of metadata (e.g. the form) should be completed by the data owners/curators.

Firstly, just move your data into the public bucket. We might need separate technical meetings if special needs are required; otherwise, we will harvest the data into Dataverse based on the responses you fill in the spreadsheet.

For sure it will be helpful. The EnviStor support team will work closely with data analysts and data owners to customize to their desired computing environment for the repository. The more knowledge or expertise the domain data analysts can provide, the better.

Yes — EnviStor runs its own self-hosted Gitea instance, giving the team full control over source hosting, access, and integration with the rest of the platform. Teams get real Git hosting for active development, code review, and release history, not just a place to drop a zip file.

Gitea is wired directly into the platform's automation: it drives our Git-as-CMS static sites and triggers Coolify-based deployments through webhooks, so code changes, deployments, and provenance stay linked end-to-end.

Alongside Git hosting, the rest of the reproducibility model still applies:

• Gitea is for active code development, review, and internal collaboration — self-hosted, full control, integrated with the platform.
• Containers encapsulate each data owner's environment (scripts, workflows, dependencies) for consistent execution across local development, OSPool, and Raptor, avoiding dependency drift.
• Dataverse archives and documents what was used to generate a research output — a snapshot of the code as a citable research artifact, distinct from the live, actively developed repository on Gitea.

This layered model keeps active development, portable execution, and long-term archival separate but connected, supporting traceability and compliance with FAIR principles.