You definitely need a program that does managed file transfer. I was in the same boat with the company I work for last year. We had 8 different servers/glorified desktops running various scripts and data massaging to make csv and xml files. The actual FTP was manually done and was not secure either. It wasn't till after this process was pieced together that a new manager developed a software approval process. It seemed rigid at first, but we could still download and try things in development. Overall the approved software list has made life easier while our company was expanding, but yes it has made getting "good" applications harder to get in production.
I will try address your concerns while trying not to sound like a running ad for the solution we ended up selecting. We tried 4 different solutions and are so happy with the software we decided on.
1) Installation Complexity. Your application should install with an installer and regardless of platform, should have minimal pre-requisites. It will be good to figure out from the solution if upgrades are provided, how are they installed and can they be rolled back. We have AS400's, Linux, and Windows servers. The GoAnywhere Director application we went with installs the same way on each platform, upgrades the same, and since it's all administered through a secure web-browser, there is no client software! If you have applications that need to programatically call different projects or routines, GoAnywhere has Run Project API's that you can place in your applications.
2) Support. You can't pay enough for good support and for people that understand your business and what needs to get done. When we were looking for an application we went from companies who were off-shore and couldn't get things to work to Linoma Software, the makers of GoAnywhere, who are in the mid-west, and know how to use both the systems and the software. They've held our hands, showed us how to make projects work with variables and make some complex business process that can be designed with a GUI. If you understand the programming side of what you want to do, the sky is the limit and they even support that.
3) Managed File Transfer solutions are not cheap. But when we realized the cost of the application and annual support versus what we were paying in manhours and the risk of having so many points of failure along with the sensitivity of the information, we estimate our ROI was 3 months and now we are saving 20 hours a month and running on 3 servers that are all locked up in a data center.
4) Error handling. We used to know if something broke because the recipients would complain they did not receive the deliverable. A solution like GoAnywhere has error handling built in at multiple levels with easy to understand logs and it will even email you to tell you what is happening. So now we have if/then handling built in to projects that if something fails like a file is missing, then the project can either notify and stop or it can go with what it has or set variables that will trigger different processes.
5) Every application has a way of doing things, but trying to learn a variant of some language to perform the same task is not a good use of time. The Director program is browser-based, click links to build processes, and it just works. What has been great is the online help, video tutorials, and user guide for this application.
6) Managing server resources. No application is helpful if it lags a server. Again we had 8 machines doing things before b/c not one application did it all and some would bring a production machine to its knees. With GoAnywhere it can run multiple threads or processes simultaneously without making the server sweat.
7) Temporary files and bits and pieces all over the place is not a way to maintain data security. Unless your systems is locked tight with user roles too many people can get to those temp files and reconstruct way too much. With a good application like this GoAnywhere Director, we can read in huge files, perform all kinds of functions, and write out encrypted results on the fly with no temporary bits and pieces or slave files.
8) Operating time - different services or components take time to start and stop. GoAnywhere has everything up and available at all times, so there is not waiting
10) You could be lost in logs. We like that this solution spits out to SYSLOGS.
A few things you didn't cover:
1) Role based access - you need to have role-based access. No one person should have free reign. GoAnywhere is based on this and does it well. PCI and HIPAA require it.
2) Fail-over, expandability, future-proofing - we run our databases on a central DB server. not too many applications support running on an external database, but for maximum uptime we are happy that we are good in this area with GoAnywhere.
Here are some links that you might find helpful: