How to grab your data from Russia in current situation

You have some infrastructure in Russia and you want to move out, for example to Europe? Now is it a problem. During last tests Russia limited long sessions, all that are longer than 1 min start throttling to 100kBps. Maybe not to all destinations, but I checked many major Europe providers and Belarus as proxy, BTW, Belarus throttling 50kBps.

As one of solutions can be torrents. But I can't use it and don't have time to test it. My solution:

First test, simple scp direct to my server in Europe - 100kBps. Rsync using a server, also 100kBps. But at the same time I found - speedtest show good speed 15-20MBps and inside Russia up to 100MBps. So, http protocol still works. And you can use any shared storage like S3, GoogleDrive, DropBox… with better speed, 1-2MBps. I use rclone.
root@box:/mnt# rclone --transfers 1 --stats 10s --stats-unit bits sync /mnt/test.txt s3:bucketname/test/ 2>&1| grep '% done'
 *     test.txt:  2% done, 117.282 MBits/s, ETA: 1m8s
 *     test.txt:  2% done, 60.200 MBits/s, ETA: 2m12s
 *     test.txt:  2% done, 30.900 MBits/s, ETA: 4m18s
 *     test.txt:  2% done, 15.861 MBits/s, ETA: 8m23s
 *     test.txt:  2% done, 8.141 MBits/s, ETA: 16m21s
 *     test.txt:  2% done, 6.291 MBits/s, ETA: 21m3s
 *     test.txt:  3% done, 5.342 MBits/s, ETA: 24m41s
 *     test.txt:  3% done, 2.742 MBits/s, ETA: 48m5s
 *     test.txt:  3% done, 1.407 MBits/s, ETA: 1h33m41s
 *     test.txt:  3% done, 739.771 kBits/s, ETA: 3h2m31s
 *     test.txt:  3% done, 379.718 kBits/s, ETA: 5h55m36s
 *     test.txt:  3% done, 2.604 MBits/s, ETA: 50m22s
 *     test.txt:  3% done, 1.337 MBits/s, ETA: 1h38m8s
 *     test.txt:  4% done, 2.416 MBits/s, ETA: 54m2s
 *     test.txt:  4% done, 1.240 MBits/s, ETA: 1h45m16s
 *     test.txt:  4% done, 651.722 kBits/s, ETA: 3h25m5s
 *     test.txt:  4% done, 334.522 kBits/s, ETA: 6h39m34s
 *     test.txt:  4% done, 2.426 MBits/s, ETA: 53m31s
 *     test.txt:  4% done, 1.245 MBits/s, ETA: 1h44m17s
 *     test.txt:  5% done, 3.220 MBits/s, ETA: 40m7s
 *     test.txt:  5% done, 1.653 MBits/s, ETA: 1h18m10s
 *     test.txt:  5% done, 868.670 kBits/s, ETA: 2h32m18s
 *     test.txt:  5% done, 445.879 kBits/s, ETA: 4h56m43s
 *     test.txt:  5% done, 2.072 MBits/s, ETA: 1h2m1s
 *     test.txt:  5% done, 1.064 MBits/s, ETA: 2h0m49s
 *     test.txt:  5% done, 559.116 kBits/s, ETA: 3h55m24s
 *     test.txt:  6% done, 2.257 MBits/s, ETA: 56m39s
 *     test.txt:  6% done, 1.158 MBits/s, ETA: 1h50m23s
 *     test.txt:  6% done, 608.803 kBits/s, ETA: 3h35m4s
 *     test.txt:  6% done, 2.281 MBits/s, ETA: 55m45s
 *     test.txt:  6% done, 1.171 MBits/s, ETA: 1h48m37s
 *     test.txt:  6% done, 615.521 kBits/s, ETA: 3h31m36s
 *     test.txt:  6% done, 315.940 kBits/s, ETA: 6h52m16s
 *     test.txt:  7% done, 1.888 MBits/s, ETA: 1h7m1s
 *     test.txt:  7% done, 992.312 kBits/s, ETA: 2h10m34s
 *     test.txt:  7% done, 2.610 MBits/s, ETA: 48m13s
...
But after a long time, a few hours, the speed will degrade to 100kBps too. One of the best solutions, split your data to 10MB pieces.
split -b 10M ./test.txt "test.txt.part."

root@box:/mnt# ls -lah | tail -10 
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.dp
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.dq
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.dr
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.ds
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.dt
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.du
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.dv
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.dw
-rw-r--r--  1 root root  10M Mar  9 13:42 test.txt.part.dx
-rw-r--r--  1 root root 4.0M Mar  9 13:42 test.txt.part.dy
And every new 10MB file will start at a good speed. In my case I have stable 2-4MBps for all time (almost one week).
rclone --transfers 2 --stats 2s --stats-unit bits sync /mnt/parts/ s3:bucketname/test/parts/

...
Transferred:    20 MBytes (3.313 MBits/s)
Transferred:    40 MBytes (4.552 MBits/s)
Transferred:    60 MBytes (3.326 MBits/s)
...
After, just gluing, and that all.
root@box:/mnt/parts# cat test.txt.part.* >> test.txt
root@box:/mnt/parts# md5sum test.txt ../test.txt 
4b7b03eced07458c98465e0b3cc694dc  test.txt
4b7b03eced07458c98465e0b3cc694dc  ../test.txt

Comments

Popular posts from this blog

FreeRadius and Google Workspace LDAP

pssh (parallel-ssh) problems on Debian 10 with Python 3.7