We recently added the capability to download logs for an entire group of tasks. Groups can include embedded runs, parallel tasks, or calls to packages.
Asynchronous User Experience
At first, we thought we were going to have to build an asynchronous user experience commonly used for batch downloads. We have some customers who run hundreds of parallel tasks, and with logs containing several megabytes, an archive for an entire group could be quite large.
The common UX for this is:
- User clicks to download an archive of logs
- System displays a message indicating that the archive is being prepared
- Archive is built via a background process
- Once archive is complete, notify the user that the download is ready
- User can click to download the archive
RWX is a platform built on performance though, so we wanted to figure out how to make the UX faster.
Immediate Download User Experience
Part of optimizing our platform is doing as much work as possible on the fly. For example, when processing task outputs, we calculate file system hashes while we're uploading.
We realized that although we needed to build an archive of all of the log files, we could build it on the fly as we're streaming the contents to the user.
This user experience is much better:
- User clicks to download an archive of logs
- Download starts immediately
- As contents are being transferred, the system pushes log files into the archive
Demo
Related posts
Read more on updates and advice from the RWX engineering team
We deleted our Dockerfiles: a better, faster way to build container images
Two weeks ago, we deleted the Dockerfile for our application, and we deleted the step in our CI pipelines that previously ran docker build.
