Changes between Initial Version and Version 1 of Ticket #2309, comment 28


Ignore:
Timestamp:
Mar 24, 2018, 9:46:54 PM (7 years ago)
Author:
tyler.haske

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #2309, comment 28

    initial v1  
    1313A lighter paper, taking into account buffer sizes, BDP tuning and multiple streams still found that ... the fastest transfers all used multiple streams:
    1414
     15Balancing TCP Buffer vs Parallel Streams in Application Level Throughput Optimization
     16
    1517https://www.cse.buffalo.edu/faculty/tkosar/papers/dadc_2009.pdf
    1618
    1719I was looking for other file transfer methods and I ran across this which seems to show the current state-of-the-art for moving petabytes of data:
    1820
     21How to transfer large amounts of data via network by Harry Mangalam
     22
    1923http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html#_parallel_tcp
    2024
    21 Apparently Google has known about this for some time, they wrote their own utility for copying files to their cloud, "gsutil". "rclone" is the app for other cloud providers, and it uses Parallel TCP too.
     25Apparently, Google has known about this for some time, they wrote their own utility for copying files to their cloud, "gsutil". "rclone" is the app for other cloud providers, and it uses Parallel TCP too.
    2226
    2327In the Grid Computing space where research institutions move petabytes of files, the most used app is GridFTP ... which as you can guess, supports parallel TCP.