Rsync Simplified

Rsync is an extremely versatile, powerful, and just downright useful tool to be have. Being able to utilize it properly is extraordinarily useful and will save you hours of work. I have yet to find a good quick-setup guide to get started with basic rsync functionality so here we go.

About Rsync
Rsync is an extremely efficient way to keep data synced between two locations. It works by synchronizing the differences of files instead of copying the whole files over. With rsync one machine acts as the server (which runs the daemon) and the other machines act as clients (they have no configuration associated with them and can sync files just by issuing the rsync command)

Rsync is probably already installed if you use Linux (if not, sudo aptitude install rsync).

Server Config
To start the rsync daemon:
           # rsync --daemon

To stop the rsync daemon:
           # killall rsync

In order for the server to know what to do, we need a config file located at:
          /etc/rsyncd.conf

Here is how mine looks (copy this if you want to get up and running quickly)

 motd file = /etc/rsyncd.motd  
 log file = /var/log/rsyncd.log  
 pid file = /var/run/rsyncd.pid  
 lock file = /var/run/rsync.lock  
 [NAME]  
   path = /home/<USERNAME>/Desktop/Storage/rsync #this is just an example path 
   comment = This is a comment
   uid = nobody  
   gid = nogroup #nogroup is for debian, "nobody" will work for most others  
   read only = no  
   list = yes  
   auth users = username #any username you want, just make sure to remember it  
   hosts allow = 192.168.10.0/24  #this is for allowing access only to your local network
   secrets file = /etc/rsyncd.scrt  

Now run these commands:
       # chmod 755 -R /home/<USERNAME>/Desktop/storage/rsync
       # chown -R nobody /home/<USERNAME>/Desktop/storage/rsync
       # chmod 600 /etc/rsyncd.scrt

Next we need to add the user we just used for our new setup to /etc/rsyncd.scrt. Edit /etc/rsyncd.scrt so that it looks like this (replace username and password with the actual info)

 username:password  

Now in order to have rsync start every boot edit /etc/init.d/rsync and change RSYNC_ENABLE=false to true.
Done! The server is now ready to go, just run rsync --daemon to start up the server proccess

Client Command
This is probably the more difficult part of the setup, and that is that you need to know how to construct your rsync command properly. Here are some things to think about:

• Am I going to keep the user permission settings (ie. can only be read/written by root)?
• Am I going to include backup files (.bak)?
• Am I going to include sub folders?
• Am I going to include file links?

Here are some common settings you may want to use:
--verbose
--progress
--stats
--compress
--rsh=/usr/local/bin/ssh
--recursive
--times
--perms
--links
--delete
--exclude "*bak"

(Notes: delete syncs deletes across syncs ie. I delete a file and sync so it deletes the file on the server. rsh is for encrypting traffic. compress compresses for transit purposes)

Ok, so that list can actually be drastically simplified for most uses (but its good to know what you're doing),
here are the shortcuts:
-a (this combines recursive, symlinks, perms)
-v (verbose)
-h (hardlinks)
-z (compress)
-e ssh (use encryption)

Now take those and put them together based on your needs. Here is how mine looks (syncing pictures on one computer to another):

rsync -avhz --progress --delete /home/user/Pictures/ user@remote.host::NAME/Pictures/

That's it!

DEBUG:
If you get any auth error (or other) here are some things to try-

• make sure you ran chmod 600 and chown nobody and chmod 755 as stated in the server setup instructions

• -e ssh requires a user on the machine with that username (test it by leaving it off to see if it works)
• with: "user@remote.host::NAME/Pictures/" pay attention to NAME which is from  the rsyncd.conf in the brackets ("[NAME]") as well as the :: (double colons). try both single ":" and double ":" if you are having trouble.
• Worst case scenario, just use a user that is already on the machine and set the settings to their credentials (not optimal but has a much higher likelihood of working)

Increase your productivity by increasing screen space

It has come to my attention that some people out there (like Peter Legierski) believe that having a bigger monitor and/or more monitors is distracting and does not increase productivity. They are wrong. And here is why:

Scenario - You are developing a web application using Node.js and MongoDB. Here is the list of windows you should have open:

1. IDE (I like eclipse but most others work just as well)
2. Web browser for API documentation
3. Web browser for live testing and/or CSS editing (Chrome is amazing for live editing CSS *will try Firefox soon, here is an article detailing the process)
4. MongoDB terminal instance open to make sure the database is functioning properly
5. Node.js terminal instance open to debug and make sure things run smoothly and so that you are alerted when things go bad
6. File browser for quickly moving files around
7. CPU, Network, and RAM monitor - this is so that if one of your apps goes haywire and starts blocking a whole thread (and since you have 8 cores you don't notice immediately) you can react quickly. *note that it may suffice to just have the monitoring widgets in your Gnome2 panel (Gnome2 FTW)
8. Working terminal for installing dependencies on the fly and for all your other Linux needs

As an experienced web developer I can say that none of these windows are optional if you want development to go as smoothly as possible (except for the web browser for api/documentation). Not all of them have to be visible at once, but being able to see more at the same time allows you to react more readily when things start behaving erratically. These windows are not distractions, they are more like a sixth sense if you will (btw, you already have more than 5 senses). They do not attract attention unless they need to. My view is only slightly skewed by the fact that since I have 5 monitors I can have all of these windows visible at the same time.

(note: Alt-Drag is the easiest way to move windows around, middle click sends windows to the back, Ctrl-W closes most windows, Ctrl-D closes terminal windows) - Debian Squeeze

Open Source Software

I wrote a research paper about open source software, and how it is the best. Here it is:
https://docs.google.com/document/pub?id=1t_Cq-25pGlQlGNDqU9wMPAxuQGhE_IsUlOkSQYv1prc
Its 2300 words long (3400 including bibliography). Also I tried not to attack Microsoft too much and tried to focus on positive things.

ASCII character naming

After looking at an article on coding horror, I have decided to post the names I use unusual ASCII characters I use. (in order of least known to most known ish)

- - tak
' - tik
| - pipe
! - bang
. - dot
~ - squiggly
% - mod
^ - up carrot
& - ampersand
* - star
( - open paren
) - close paren
` - accent
@ - at
# - hash
$ - dollar sign
_ - underscore
= - equals
" - quote
/ - slash
? - question mark
\ - back slash
> - less than
< - greater than
, - comma
{ - open brace
} - close brace
[ - open bracket
] - close bracket
: - colon
; - semi colon

5 Monitors

Yup... That's right, I now have 5 monitors. The setup was easy, just had to fix 1 thing, and that was to edit /etc/X11/xorg.conf and add additional space to my virtual desktop size.

*the 5th monitor IS OVERKILL, but it looks so cool. (worth it)

Mass Music Conversion and Compression

Scenario:
You have 9GB free storage on your Android smart phone (or 2GB storage on your iPhone), and you have 14GB of music that you want to put on it. What do you do? You have probably like 3 options:

1. Buy a micro SD card (Android only) - ~$10-20 depending on storage size
2. Only load some of your music onto your device - The cost of only having some of your music (not to mention the agony of picking the right music)
3. Compressing music, reducing sound quality but allowing you to keep all your music - The cost of running a script

Now for some, option 3 sounds absolutely ridiculous, because who want's to listen to poor-quality music? My argument  is that it's actually not that bad, and you can barely tell the difference depending on the compression level.

Lets see some before and after stats:

Before:
14.8GB
~128kb/s bitrate

After:
6.6GB
~64kb/s bitrate

Warning, if you are going to try this, try it with one song first and see if the quality is good enough for you (I tried many variations in order to reach my optimal compression quality).

How To:

1.
Forget MP3's. Yes, that's right, MP3 is TERRIBLE! (also there's licencing stuff with the codecs that gave me issues). On Ubuntu it may work out of the box, but for legal reasons Debian doesn't have the codec in its stable repositories. Luckily, I use Debian and did some research into MP3 alternative. OGG Vorbis is where it's at! "Ogg Vorbis is much better than MP3 at very low bitrates (64kbps, etc)."(wiki vs.) Alright, great now we have our audio encoding figured out.

2.
FFMPEG is your friend, play with it. Seriusly, give it a whirl and see what you can do to your audio. Here is what I used for my music:

ffmpeg -i "$input" -aq 15 -acodec vorbis -strict experimental "$output"

3.
Finally, here is the script I came up with to convert all of my music at once:

#!/bin/bash
dir="/home/user/Music/shrunken"
#the find command has a bug where it doesn't actually exclude the shrunken directory (but im sure you can fix it)
find . -path "/home/user/Music/shrunken/*" -prune -o -name *.mp3 -print| while read f
do
input=${f:2}
fileLoc=${f:1}
output="$dir${fileLoc%.*}.ogg"
extension=${output%%.*}
echo $input
folder=$(dirname "$extension")
#mkdir with the "-p" option tells it to auto-create the sub-directories required
mkdir -p "$folder"
#the "echo" is a neat trick that defaults all of the questions ffmpeg might ask about replacing files to "no"
echo "n"|ffmpeg -i "$input" -aq 15 -acodec vorbis -strict experimental "$output"
done

Yes, there are bugs. Yes, it's not perfect. But your smart, you can handle it. Good Luck!

(Side note: My music player on android is jukefox - great piece of software right there)

MATE

Finally there is a Gnome2 fork for Linux kernel 3+. Find it here. Why does this matter so much? Well, in a previous post I talked about why Gnome2 is the best desktop window manager. Linux Torvalds agrees:










Linus Torvalds on GNOME 3.0

"While you are at it, could you also fork gnome, and support a gnome2 environment?"

"I want my sane interfaces back. I have yet to meet anybody who likes the unholy mess that is gnome3."

"It's not that I have rendering problems with gnome3 (although I do have those too), it's that the user experience of Gnome3 even without rendering problems is unacceptable.

Why can't I have shortcuts on my desktop? Why can't I have the expose functionality? Wobbly windows?Why does anybody sane think that it's a good idea to have that "go to the crazy 'activities'" menu mode? I used to be upset when gnome developers decided it was "too complicated" for the user to remap some mouse buttons. In gnome3, the developers have apparently decided that it's "too complicated" to actually do real work on your desktop, and have decided to make it really annoying to do.

Here's an example of "the crazy": you want a new terminal window. So you go to "activities" and press the "terminal" thing that you've made part of your normal desktop thing (but why can't I just have it on the desktop, instead of in that insane "activities" mode?). What happens? Nothing. It brings your existing terminal to the forefront.

That's just crazy crap. Now I need to use Shift-Control-N in an old terminal to bring up a new one. Yeah, that's a real user experience improvement. Sure.

I'm sure there are other ways, but that's just an example of the kind of "head up the arse" behavior of gnome3. Seriously. I have been asking other developers about gnome3, they all think it's crazy. I'm using Xfce. I think it's a step down from gnome2, but it's a huge step up from gnome3. Really."

Linus Torvalds is a genius, so even if you don't trust me, you can at least trust him. If you'v been cursed by upgrading to kernel 3+ and being forced to use XFCE (or Gnome Shell), give MATE a try.