Tutorial: Auto Sync seedbox and home linux (and Mac!) machine with lftp shell script
Register

We are the best invite forum on the internet! Here you will find free invites, free seedboxes, free bonuses, and much more. Our members know the true meaning of sharing and have created a truly global bittorent community! Our site has the most up to date information on all private trackers and our members will guide you and introduce you to this truly secretive and enlightened club. Ready to get started? Register now!


User Tag List

Page 1 of 6 1 2 3 4 ... LastLast
Results 1 to 10 of 53
  1. #1

    Default Tutorial: Auto Sync seedbox and home linux (and Mac!) machine with lftp shell script

    I made this little script for my own needs and works like a charm, so I decided to share it.. I hope it helps someone.

    The Problem:

    So, you got a linux/bsd box at home which you use as a media and storage server, but keeping it synchronized with the files on your seedbox is a nightmare. Wouldn't it be nice if your home box downloaded the latest files on the seedbox with a sinple fire-and-forget command, or even better: did it by itself during the day while you are at work, so when you get home all the goodies are avaiable? Well, read on then.

    The requirements:

    - A machine with any flavor of Linux or BSD (does anyone still use Solaris?). Hell, it should work even on Macs, provided it has lftp built for it installed. (duh)

    Edit: I went googling for lftp on mac and found a great guide on how to install it on Linux and Mac right here in here in TI! Kudos to the hobbit! I cant test if the rest of the process will work on Mac as I typed without modification, for I lack a Mac here to test it.

    - lftp: this is a small yet hugely powerful shell ftp client. It should come default with most installations, but if your machine lack it, there's plenty of tutorials around for how to compile it from source or install binary packages (hint: if are in fedora run yum install lftp as root)

    The Solution

    - Just whip out your favorite shell text editor (should be vi, unless you are an emacs heathen) and type out the following script in any path of your choosing (for simplicity sake, I will assume from now on the path you are creating the script is /home/user, but it can be wherever you like):

    Code:
    cd /home/user
    vi synctorrents.sh
    Enter insert mode (press ' i' ), then paste the following code:

    Code:
    #!/bin/bash
    
    login=FTP_LOGIN
    pass=FTP_PASS
    host=FTP_HOST
    remote_dir=FULL_REMOTE_PATH
    local_dir=FULL_LOCAL_PATH
    
    
    trap "rm -f /tmp/synctorrent.lock" SIGINT SIGTERM
    
    if [ -e /tmp/synctorrent.lock ]
    then
      echo "Synctorrent is running already."
      exit 1
    else
    
      touch /tmp/synctorrent.lock
    
      lftp -u $login,$pass $host << EOF
      set ftp:ssl-allow no
      set mirror:use-pget-n 5
      mirror -c -P5 --log=synctorrents.log $remote_dir $local_dir
      quit
    EOF
    
      rm -f /tmp/synctorrent.lock
      exit 0
    
    fi
    Obviously enough, the value of the variables at the start of the script should be replaced with your values. I trust they are self explanatory enough.

    The important parameters i use for lftp are:
    'set mirror:use-pget-n 5' this makes lftp try to split up fies in 5 pieces for parallel downloading. Likewise,
    '-P5' means it will download at most 5 files in parallel (for a total 25 connections). Those 2 combined work wonders. In my case, I always end up downloading the files at the limit of my connection, but feel free to play with them and find what works best for you.
    '-c ' just tells it to try and resume an interrupted download if it' s the case.

    Ok, save it and close the editor (Esc, then :wq) and give it permission to run with:



    Code:
    chmod +x synctorrents.sh
    There you go! You can test it running it from where you stand with ./synctorrents.sh and he will nicely log in your seedbox' s ftp, scan the directory you configured, compare with the local directory you configured and start downloading the new or changed files!

    Now, the real nice thing about it is that you can have this bad boy in the crontab without worrying it will run over itself so go for it:

    Code:
    crontab -e
    This will open vi to your crontab file. Add the line:

    Code:
    0 * * * * /home/user/synctorrents.sh >> /home/user/sync_cron.log 2>&1
    Save and close (Esc, then :wq) and you are done!

    This will make the script run at the 0 minute of every hour (it wont run over itself if the previous instance hasn't finished running yet) and let a logfile in case anything goes wrong. Of course you can adjust any parameter of the crontab entry to run it whenever and how often you like, just google for it ;)

    You can make sure the line was really saved by checking the crontab tasks with

    Code:
    crontab -l
    Enjoy and let me know if it was useful! ;)

    By the way, if your seedbox is Linux and has lftp installed, with just a minor modification you could install a ftp server at your home (including windows) and have the seedbox pushing the files to you instead of you pulling them down. Let me know if a tutorial like that would be usefull and I will conjure up something ;)
    Last edited by LordHades; June 9th, 2011 at 12:30 PM.


  2. To remove ads become VIP. Inquire about advertising here.
  3. #2

    Default Re: Tutorial: auto Sync seedbox and home linux machine with lftp shell script

    Bit much to ask, I know, but it would be very useful if the sync is aware of any transfers of the torrentclient on the seedbox and adjusts its bandwidth accordingly.

    Lets say: if rtorrent is seeding @ 87Mbps, the script won't use more than 13Mbps for the sync.

  4. #3

    Default Re: Tutorial: auto Sync seedbox and home linux machine with lftp shell script

    Well, you can limit dandwidth use in lftp with the parameters (taken from its man page):

    net:limit-rate (bytes per second)
    limit transfer rate on data connection. 0 means unlimited. You
    can specify two numbers separated by colon to limit download and
    upload rate separately.

    net:limit-max (bytes)
    limit accumulating of unused limit-rate. 0 means twice of limit-
    rate.

    net:limit-total-rate (bytes per second)
    limit transfer rate of all connections in sum. 0 means unlim-
    ited. You can specify two numbers separated by colon to limit
    download and upload rate separately. Note that sockets have
    receive buffers on them, this can lead to network link load
    higher than this rate limit just after transfer beginning. You
    can try to set net:socket-buffer to relatively small value to
    avoid this.

    net:limit-total-max (bytes)
    limit accumulating of unused limit-total-rate. 0 means twice of
    limit-total-rate.

    These can be set the same way my original script sets mirror:use-pget-n. Then, it is just a matter of monitoring how much bandwidth your client is using. There are probably ways to query for this info from a shell script, but they would be dependent on what torrent client you use . I am open to sugestions here ;)

  5. #4

    Default Re: Tutorial: Auto Sync seedbox and home linux (and Mac!) machine with lftp shell scr

    I edited the OP to reflect the fact that I found a great lftp install guide to Linux and Mac in http://www.torrent-invites.com/seedb...linux-mac.html.
    It would be great if anyone could test the rest of my script on Mac (I don't go near those things). :)

  6. #5

    Default Re: Tutorial: auto Sync seedbox and home linux machine with lftp shell script

    Quote Originally Posted by LordHades View Post
    Then, it is just a matter of monitoring how much bandwidth your client is using. There are probably ways to query for this info from a shell script, but they would be dependent on what torrent client you use . I am open to sugestions here ;)
    Any way to set the max bandwidth in the script, like 100Mbit and then use vnstat -l to substract that from 100Mbit which leaves the remaining bandwidth for sync? And do that before every next transfer?

    Just guessing, I'm still quite new to bash scripting.

  7. #6

    Default Re: Tutorial: auto Sync seedbox and home linux machine with lftp shell script

    I thought about it, but would't the info be tainted if it is a shared seedbox? I think most of them are, so you would also be seeing other people's bandwidth use, therefore crippling your own experience based on someone else's seeding.

  8. #7

    Default Re: Tutorial: Auto Sync seedbox and home linux (and Mac!) machine with lftp shell scr

    Script works great on my Mac which receives the files from my linux seedbox.
    Thanks for posting it.
    I always liked the performance and multithreading of lftp but mostly used a gui client because they were easier.
    This script will automate most of my transfers now and I get to keep it in unix using much less resources.
    Last edited by orytek; June 11th, 2011 at 02:50 AM.

  9. #8

    Join Date
    Jun 2011
    Location
    Brisbane, Queensland, Australia
    Posts
    8

    Default Re: Tutorial: Auto Sync seedbox and home linux (and Mac!) machine with lftp shell scr

    Hi LordHades, Thanks for the in depth tutorial, The code is really clear.

    I have been trying to get rsync and scp working with mixed results but this tutorial seems to be the most promissing
    I gave it a go but ran into kinks.
    I am trying to learn linux but am having a bit of trouble.


    I have a ReadyNas Ultra Pro. I have root shell access and can access the shell.
    I also have a feralhosting seed box
    When I start the shell of the readynas it says
    Linux nas_________ 2.6.33.6.RNx86_64.2.1 #1 SMP Tue Jul 27 13:21:19 PDT 2010 x86_64 GNU/Linux


    I tried running this script it appears that lftp is not installed.
    So i downloaded the latest lftp-4.2.3.tar.gz and extracted it and tried to install it.
    However it appears that it doesn't have a C compiler ....

    Im not sure if I have any chance with this as this appears to be a very cut down version of linux?
    Maybe i can go the other way and run the script from the seedbox side as you mentioned?

    Any thoughts or ideas on what i can do would be appreciated.

    Thanks

  10. #9

    Default Re: Tutorial: Auto Sync seedbox and home linux (and Mac!) machine with lftp shell scr

    Hi rjames. Funny you mentioned rsync and scp, because I ended up turning to this solution precisely because the speed of rsync over scp wasn't cutting it :)
    But don't give up. If your NAS is just lacking a compiler, that can be installed. The de facto standard on Linux world is gcc, and you can find info on it here: GCC, the GNU Compiler Collection - GNU Project - Free Software Foundation (FSF).

    BUT: in that page you will find the source code (useless if you don't already have the compiler - duh), and digging around I didn't find a binary distribution for your platform. But on the bright side, googling for it seems your NAS has the ability to install apt-get trough a web UI (really nifty program that finds and downloads binaries for you), and I found this tutorial on setting up the compiler and C libraries it needs:

    Setting up a ReadyNAS Development Environment : NETGEAR ReadyNAS Community

    This should do the trick nicely. With a bit of luck, after having apt installed you could probably try 'apt-get install lftp' and if a package is already built on the repositories, you wont need to compile it yourself. If not, well, go with the original plan :)

    Good luck and enjoy!

    ---------- Post added at 09:40 AM ---------- Previous post was at 09:32 AM ----------

    Oh, I almost forgot. Having the mirror reversed would involve installing an ftp server on your NAS. You probably would run into a similar problem with lacking a compiler or already having it installed, plus, it is more work maintaining a server, but if you wanna try this way, the line in the main script that says

    Code:
    mirror -c -P5 --log=synctorrents.log $remote_dir $local_dir
    could be chanhed to

    Code:
    mirror -R -c -P5 --log=synctorrents.log $local_dir $remote_dir
    And run it on the seedbox. Of course, since your point of view now is the seedbox one, $local_dir would be the seedbox path, and $remote_dir your home path. The -R switched added means 'reverse mirroring', therefore, it uploads files instead of downloading.
    I hope this helps :)

  11. #10

    Default Re: Tutorial: Auto Sync seedbox and home linux (and Mac!) machine with lftp shell scr

    Please use the 'Like' button instead if you just want to say Thanks.

Page 1 of 6 1 2 3 4 ... LastLast

Similar Threads

  1. whats a seedbox and what does it do.
    By therealdeal in forum BitTorrent Discussion
    Replies: 7
    Last Post: February 23rd, 2009, 12:16 PM
  2. Problems with seedbox and private tracker
    By billybob in forum BitTorrent Discussion
    Replies: 1
    Last Post: January 23rd, 2009, 03:50 AM
  3. Seedboxes and Dedicated Servers
    By polishseed in forum Seedbox Advertising
    Replies: 0
    Last Post: January 18th, 2009, 10:11 AM
  4. Seedboxes and Dedicated Servers
    By polishseed in forum Seedbox Advertising
    Replies: 0
    Last Post: January 18th, 2009, 10:01 AM
  5. Private trackers, seedboxes and BT Junkies
    By digitalpencil in forum General Discussion
    Replies: 6
    Last Post: September 25th, 2008, 05:08 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •