EphFlow: Cross-Platform FaaS Workflows with Serverless Ephemeral VM Orchestration

Scientific workflows are essential for automating complex computational pipelines. Serverless Function-as-a-Service (FaaS) cloud platforms have the potential to enable wider adoption of workflows by simplifying programming and deployment. However, existing systems present significant adoption barriers: vendor lock-in, strict resource limitations, and limited language support. We present EphFlow, a novel open-source middleware that enables serverless execution of workflows exposing the FaaS programming abstraction while supporting different platforms, programming languages, and dynamic orchestration of ephemeral VMs, without code modifications. Its novelty includes: (1) novel middleware that supports invocation of FaaS functions within general-purpose containers that can be deployed across different platforms, abstracting platform-specific APIs and generalizing to multiple languages (including Python, R, and Julia); (2) on-demand in-DAG provisioning of ephemeral VMs that transparently overcomes FaaS resource limits for resource-intensive tasks by injecting lifecycle management actions into workflow DAGs; and (3) passive S3-based coordination that eliminates the need for dedicated, managed workflow engines. Experiments demonstrate deployment of unmodified workflow DAGs and multi-language FaaS functions across five different platforms (GitHub Actions, AWS Lambda, Google Cloud Run, OpenWhisk, and SLURM) for synthetic workflows, micro-benchmarks, and an event-driven ecological forecasting application (FLARE). Experiments also demonstrate the ability to deploy ephemeral VMs via provisioning of GitHub self-hosted Runners on AWS EC2. Experiments with a micro-benchmark workflow deployed across Lambda, Google Cloud, and GitHub Actions with 1,051 invocations quantifies key performance metrics: cross-platform, multi-language FaaS entry point handling (0.3-5s for payload processing and user function invocation) and latency between action invocations (median of 9.3s in Lambda, 131.0s with EphFlow VMs).

Recommended citation: Ashish Ramrakhiani, Nolan Cutler, R. Quinn Thomas, Cayelan C. Carey, Austin Delany, Izzy Lerman, Owen Wickman, Renato J. Figueiredo, "EphFlow: Cross-Platform FaaS Workflows with Serverless Ephemeral VM Orchestration", 22nd IEEE International Conference on e-Science, 2026. https://ieeexplore.ieee.org/xpl/conhome/1001511/all-proceedings