Five years of production is a better designer than any whiteboard

FTP2SF has been running in production for five years now. During that time the platform has moved a great many files, connected Salesforce to a long list of external systems and, rather more importantly, accumulated enough real-world use to show us where the original architecture could be improved.

We have recently spent a fair amount of time doing exactly that. We have reworked the authentication layer, rebuilt the client-side controls and changed the way files are transferred through the user interface. None of that is particularly interesting if you read it as a list of product features. It becomes considerably more interesting when you look at why we did it.

The easiest mistake in serverless is putting the server in the middle of everything

The problem with a lot of serverless applications is that it is very easy to end up putting the server in the middle of everything. A browser makes a request, Lambda processes it, Lambda retrieves something from another service, and then Lambda sends it back to the browser. For a small transaction that is perfectly reasonable. For a large file it is often just unnecessary plumbing, and you pay for that plumbing in execution time, in memory, and in the patience of whoever clicked the button.

So we changed the way FTP2SF handles file transfers. Where the underlying provider supports it, FTP2SF can now establish a signed transfer and allow the browser to stream the file directly from the provider. The same approach works for uploads. The Lambda layer is still very much involved in the operation, but it is no longer required to physically carry every byte of the file.

That distinction is important. Lambda remains responsible for the things that genuinely should be controlled by the application: authentication, authorisation, validation, signing and the orchestration of the operation. The browser is simply given a narrowly scoped ability to perform a transfer it has already been authorised to perform.

That makes a considerable difference to the experience. A large download no longer needs to make a round trip through the application simply because somebody clicked a button in the FTP2SF interface, and the same applies to uploads. The application still controls what is allowed to happen. The data itself can take the most appropriate path. We have written before about where Lambda earns its place and where it quietly costs you, and this is that same argument applied to the part of the system that moves the most bytes.

Authentication had to stop being a per-operation detail

The other significant piece of work has been authentication. FTP2SF now has considerably more interaction between the client, the application services and external providers than it did when we originally built it. The authentication model therefore needed to be treated as a proper platform capability rather than as something implemented independently around individual operations.

The new layer gives us a cleaner separation between authentication, authorisation and access to the underlying resource. That matters a great deal more than it used to, because we are now issuing signed transfer capabilities to a browser. A browser should not receive broad credentials to an external system simply because it needs to download one file. It should receive precisely the access required for that particular operation, for the period in which it is required, and nothing beyond that.

Anyone who has followed the product for a while will recognise this as a continuation of the work we did on how FTP2SF authenticates against SFTP. The direction is the same: make the access model something the platform owns and can reason about, rather than something each operation reinvents on its own terms.

Moving from Aura to LWC, and taking the opportunity

We also finally completed the move from the original Aura implementation to Lightning Web Components. That sounds like a fairly routine Salesforce technology upgrade, but we took the opportunity to do rather more than reproduce the existing interface in a newer framework.

The old UI had naturally evolved around the way FTP2SF was originally delivered. Moving to LWC let us properly componentise the individual capabilities of the file management experience. Selecting a site, navigating folders and directories, and managing files can now be treated as independent components rather than being welded together as one large piece of interface.

That gives us considerably more flexibility in how FTP2SF is presented inside Salesforce. The components can be bundled together to provide the complete file management experience, or used independently where a customer only needs part of it. A page can have site selection and navigation without file management. File management can be embedded into an existing experience. Individual components can simply be left out where they are not relevant.

We also changed the way file previews work. Rather than treating the preview as an inseparable part of the FTP2SF interface, it is now a customisable component in its own right. Customers can decide what should happen when a file is opened and define the experience they want around that file type, which is something we have been asked for often enough to know it was never a niche request.

For us this is a far more useful outcome than simply having a newer Salesforce UI. The underlying capabilities can be assembled into different experiences depending on what a customer is actually trying to achieve, without anybody having to change the underlying application.

Five years in, we are still making it faster

There is a tendency in software to equate progress with adding things. More endpoints, more integrations, more configuration, more features. Five years of running FTP2SF has reinforced a different view. Sometimes the best engineering work available to you is removing unnecessary work from a system that already does what it is supposed to do.

FTP2SF already worked. These changes are about making it work more efficiently, making the security model cleaner, and making the client experience considerably faster without throwing away an architecture that has been running in production for years. That strikes me as a much more useful definition of an upgrade than adding another item to a feature list.

• FTP2SF and SFTP Authentication
• Secure File Transfer