We have to move a set of very large data (in petabytes) from HPC cluster to a storage server. We have a high capacity communication link between the devices. However, the bottleneck seems to be a fast transfer tool that can be parallelized for individual files (because the individual files are each in terabytes).
In this regard, I am looking for a tool that does not require admin rights and is still considerably faster than scp or rsync. If there is any tool that can be installed locally without admin rights, that will also be useful. I come across this link (What is the fastest way to send massive amounts of data between two computers?), which mentions the netcat way but we couldn't make it work.
For information, we are trying to copy relatively few files with very large size (and not many many small files).
Appreciate your time and help