Sunday, June 17, 2012

Rsync Simplified


Rsync is an extremely versatile, powerful, and just downright useful tool to be have. Being able to utilize it properly is extraordinarily useful and will save you hours of work. I have yet to find a good quick-setup guide to get started with basic rsync functionality so here we go.

About Rsync
Rsync is an extremely efficient way to keep data synced between two locations. It works by synchronizing the differences of files instead of copying the whole files over. With rsync one machine acts as the server (which runs the daemon) and the other machines act as clients (they have no configuration associated with them and can sync files just by issuing the rsync command)

Rsync is probably already installed if you use Linux (if not, sudo aptitude install rsync).

Server Config
To start the rsync daemon:
           # rsync --daemon

To stop the rsync daemon:
           # killall rsync

In order for the server to know what to do, we need a config file located at:
          /etc/rsyncd.conf

Here is how mine looks (copy this if you want to get up and running quickly)

 motd file = /etc/rsyncd.motd  
 log file = /var/log/rsyncd.log  
 pid file = /var/run/rsyncd.pid  
 lock file = /var/run/rsync.lock  
 [NAME]  
   path = /home/<USERNAME>/Desktop/Storage/rsync #this is just an example path 
   comment = This is a comment
   uid = nobody  
   gid = nogroup #nogroup is for debian, "nobody" will work for most others  
   read only = no  
   list = yes  
   auth users = username #any username you want, just make sure to remember it  
   hosts allow = 192.168.10.0/24  #this is for allowing access only to your local network
   secrets file = /etc/rsyncd.scrt  

Now run these commands:
       # chmod 755 -R /home/<USERNAME>/Desktop/storage/rsync
       # chown -R nobody /home/<USERNAME>/Desktop/storage/rsync
       # chmod 600 /etc/rsyncd.scrt

Next we need to add the user we just used for our new setup to /etc/rsyncd.scrt. Edit /etc/rsyncd.scrt so that it looks like this (replace username and password with the actual info)

 username:password  


Now in order to have rsync start every boot edit /etc/init.d/rsync and change RSYNC_ENABLE=false to true.
Done! The server is now ready to go, just run rsync --daemon to start up the server proccess

Client Command
This is probably the more difficult part of the setup, and that is that you need to know how to construct your rsync command properly. Here are some things to think about:

  • Am I going to keep the user permission settings (ie. can only be read/written by root)?
  • Am I going to include backup files (.bak)?
  • Am I going to include sub folders?
  • Am I going to include file links?
Here are some common settings you may want to use:
--verbose
--progress
--stats
--compress
--rsh=/usr/local/bin/ssh
--recursive
--times
--perms
--links
--delete
--exclude "*bak"

(Notes: delete syncs deletes across syncs ie. I delete a file and sync so it deletes the file on the server. rsh is for encrypting traffic. compress compresses for transit purposes)

Ok, so that list can actually be drastically simplified for most uses (but its good to know what you're doing),
here are the shortcuts:
-a (this combines recursive, symlinks, perms)
-v (verbose)
-h (hardlinks)
-z (compress)
-e ssh (use encryption)

Now take those and put them together based on your needs. Here is how mine looks (syncing pictures on one computer to another):

rsync -avhz --progress --delete /home/user/Pictures/ user@remote.host::NAME/Pictures/

That's it!

DEBUG:
If you get any auth error (or other) here are some things to try-

  • make sure you ran chmod 600 and chown nobody and chmod 755 as stated in the server setup instructions
  • -e ssh requires a user on the machine with that username (test it by leaving it off to see if it works)
  • with: "user@remote.host::NAME/Pictures/" pay attention to NAME which is from  the rsyncd.conf in the brackets ("[NAME]") as well as the :: (double colons). try both single ":" and double ":" if you are having trouble.
  • Worst case scenario, just use a user that is already on the machine and set the settings to their credentials (not optimal but has a much higher likelihood of working)


Friday, May 25, 2012

Increase your productivity by increasing screen space

It has come to my attention that some people out there (like Peter Legierski) believe that having a bigger monitor and/or more monitors is distracting and does not increase productivity. They are wrong. And here is why:

Scenario - You are developing a web application using Node.js and MongoDB. Here is the list of windows you should have open:

  1. IDE (I like eclipse but most others work just as well)
  2. Web browser for API documentation
  3. Web browser for live testing and/or CSS editing (Chrome is amazing for live editing CSS *will try Firefox soon, here is an article detailing the process)
  4. MongoDB terminal instance open to make sure the database is functioning properly
  5. Node.js terminal instance open to debug and make sure things run smoothly and so that you are alerted when things go bad
  6. File browser for quickly moving files around
  7. CPU, Network, and RAM monitor - this is so that if one of your apps goes haywire and starts blocking a whole thread (and since you have 8 cores you don't notice immediately) you can react quickly. *note that it may suffice to just have the monitoring widgets in your Gnome2 panel (Gnome2 FTW)
  8. Working terminal for installing dependencies on the fly and for all your other Linux needs
As an experienced web developer I can say that none of these windows are optional if you want development to go as smoothly as possible (except for the web browser for api/documentation). Not all of them have to be visible at once, but being able to see more at the same time allows you to react more readily when things start behaving erratically. These windows are not distractions, they are more like a sixth sense if you will (btw, you already have more than 5 senses). They do not attract attention unless they need to. My view is only slightly skewed by the fact that since I have 5 monitors I can have all of these windows visible at the same time.

(note: Alt-Drag is the easiest way to move windows around, middle click sends windows to the back, Ctrl-W closes most windows, Ctrl-D closes terminal windows) - Debian Squeeze

Wednesday, May 23, 2012

Open Source Software

I wrote a research paper about open source software, and how it is the best. Here it is:
https://docs.google.com/document/pub?id=1t_Cq-25pGlQlGNDqU9wMPAxuQGhE_IsUlOkSQYv1prc
Its 2300 words long (3400 including bibliography). Also I tried not to attack Microsoft too much and tried to focus on positive things.

Sunday, May 20, 2012

ASCII character naming

After looking at an article on coding horror, I have decided to post the names I use unusual ASCII characters I use. (in order of least known to most known ish)

- - tak
' - tik
| - pipe
! - bang
. - dot
~ - squiggly
% - mod
^ - up carrot
& - ampersand
* - star
( - open paren
) - close paren
` - accent
@ - at
# - hash
$ - dollar sign
_ - underscore
= - equals
" - quote
/ - slash
? - question mark
\ - back slash
> - less than
< - greater than
, - comma
{ - open brace
} - close brace
[ - open bracket
] - close bracket
: - colon
; - semi colon

Tuesday, May 1, 2012

5 Monitors


Yup... That's right, I now have 5 monitors. The setup was easy, just had to fix 1 thing, and that was to edit /etc/X11/xorg.conf and add additional space to my virtual desktop size.

*the 5th monitor IS OVERKILL, but it looks so cool. (worth it)

Wednesday, April 18, 2012

Mass Music Conversion and Compression


Scenario:
You have 9GB free storage on your Android smart phone (or 2GB storage on your iPhone), and you have 14GB of music that you want to put on it. What do you do? You have probably like 3 options:
  1. Buy a micro SD card (Android only) - ~$10-20 depending on storage size
  2. Only load some of your music onto your device - The cost of only having some of your music (not to mention the agony of picking the right music)
  3. Compressing music, reducing sound quality but allowing you to keep all your music - The cost of running a script
Now for some, option 3 sounds absolutely ridiculous, because who want's to listen to poor-quality music? My argument  is that it's actually not that bad, and you can barely tell the difference depending on the compression level.

Lets see some before and after stats:
Before:
14.8GB
~128kb/s bitrate

After:
6.6GB
~64kb/s bitrate

Warning, if you are going to try this, try it with one song first and see if the quality is good enough for you (I tried many variations in order to reach my optimal compression quality).

How To:

1.
Forget MP3's. Yes, that's right, MP3 is TERRIBLE! (also there's licencing stuff with the codecs that gave me issues). On Ubuntu it may work out of the box, but for legal reasons Debian doesn't have the codec in its stable repositories. Luckily, I use Debian and did some research into MP3 alternative. OGG Vorbis is where it's at! "Ogg Vorbis is much better than MP3 at very low bitrates (64kbps, etc)."(wiki vs.) Alright, great now we have our audio encoding figured out.

2.
FFMPEG is your friend, play with it. Seriusly, give it a whirl and see what you can do to your audio. Here is what I used for my music:
ffmpeg -i "$input" -aq 15 -acodec vorbis -strict experimental "$output"

3.
Finally, here is the script I came up with to convert all of my music at once:


#!/bin/bash
dir="/home/user/Music/shrunken"
#the find command has a bug where it doesn't actually exclude the shrunken directory (but im sure you can fix it)
find . -path "/home/user/Music/shrunken/*" -prune -o -name *.mp3 -print| while read f
do
input=${f:2}
fileLoc=${f:1}
output="$dir${fileLoc%.*}.ogg"
extension=${output%%.*}
echo $input
folder=$(dirname "$extension")
#mkdir with the "-p" option tells it to auto-create the sub-directories required
mkdir -p "$folder"
#the "echo" is a neat trick that defaults all of the questions ffmpeg might ask about replacing files to "no"
echo "n"|ffmpeg -i "$input" -aq 15 -acodec vorbis -strict experimental "$output"
done
Yes, there are bugs. Yes, it's not perfect. But your smart, you can handle it. Good Luck!

(Side note: My music player on android is jukefox - great piece of software right there)

Friday, April 6, 2012

MATE

Finally there is a Gnome2 fork for Linux kernel 3+. Find it here. Why does this matter so much? Well, in a previous post I talked about why Gnome2 is the best desktop window manager. Linux Torvalds agrees:










Linus Torvalds on GNOME 3.0
"While you are at it, could you also fork gnome, and support a gnome2 environment?"

"I want my sane interfaces back. I have yet to meet anybody who likes the unholy mess that is gnome3."

"It's not that I have rendering problems with gnome3 (although I do have those too), it's that the user experience of Gnome3 even without rendering problems is unacceptable.

Why can't I have shortcuts on my desktop? Why can't I have the expose functionality? Wobbly windows?Why does anybody sane think that it's a good idea to have that "go to the crazy 'activities'" menu mode? I used to be upset when gnome developers decided it was "too complicated" for the user to remap some mouse buttons. In gnome3, the developers have apparently decided that it's "too complicated" to actually do real work on your desktop, and have decided to make it really annoying to do.

Here's an example of "the crazy": you want a new terminal window. So you go to "activities" and press the "terminal" thing that you've made part of your normal desktop thing (but why can't I just have it on the desktop, instead of in that insane "activities" mode?). What happens? Nothing. It brings your existing terminal to the forefront.

That's just crazy crap. Now I need to use Shift-Control-N in an old terminal to bring up a new one. Yeah, that's a real user experience improvement. Sure.

I'm sure there are other ways, but that's just an example of the kind of "head up the arse" behavior of gnome3. Seriously. I have been asking other developers about gnome3, they all think it's crazy. I'm using Xfce. I think it's a step down from gnome2, but it's a huge step up from gnome3. Really."



Linus Torvalds is a genius, so even if you don't trust me, you can at least trust him. If you'v been cursed by upgrading to kernel 3+ and being forced to use XFCE (or Gnome Shell), give MATE a try.

Tuesday, April 3, 2012

Apple... Stop Hatin'

Apple... Why? As a developer I see the benefits of using OSX as a development platform (at least over Windows), what I can't get over is the fact that I can't develop iPhone applications under Linux. I don't mind paying the $100 and going through the review process, what I do mind is being restricted to an inferior operating system on inferior hardware. In my first post I wrote about my setup, 8-Cores 8GB DDR3 1866GHz RAM, and the Ati Radeon HD 5870. I didn't mention that my build came in at ~$1000 (I'v made upgrades here and there but 1k is a good estimate.)

Now lets compare my ~$1000 computer to a similarly priced Apple product. The iMac, $1200.

  • 4GB RAM
  • Quad Core (2.5GHz)
  • Ati Radeon HD 6750M


What... Those specs don't even come close to mine. Yes, I can buy a Mac Pro, and yes I know that I'm paying for the brand as well (+OS). But in order to stay competitive on the hardware side of things with regards to developing, I can't be spending $2000+ every other year for an up-to-date machine.

Apple, you need to open up to Linux developers (especially for the iPhone).

Monday, February 13, 2012

xmonad and DWM

Recently I removed Crunchbang from my net-book and installed Debian wheezy (testing). Openbox was great, but I still felt that I was relying too much on the mouse. I normally shy away from tiling window managers, mostly because I use Gnome on my desktop, but I decided to make an exception and try out xmonad.

Conclusion: xmonad (dwm in haskell) is great (and dwm is ok too, but its necessary to tweak it a bit so that it looks nice)

Now I say this because xmonad allows me to stay on the keyboard for pretty much everything (besides web browsing - and there's a way to even do that, though I won't be trying it anytime soon). Accessing multiple work-spaces is as easy as <Alt>+<Work-space Number>. If I need to float a window, its as easy as dragging it while holding the <Alt> key.

So without further ado, lets go through some setup steps.

Step 1. Read the tutorial, Its short and really easy to get a hold of

Step 2. Install xmonad (aptitude install xmonad)

Step 3. Install dmenu (aptitude install xmonad)

Step 4. Ok, so here's where I probably screwed up big time, but hey it works. So basically I need a GUI for wireless (too much encryption junk to remember for commandline), and a way to monitor battery life.
The tools I decided to use were trayer (a tray like gnome-panel to hold nm-applet) and nm-applet. This allowed me easily connect to wireless. Next i needed to monitor battery (and actually a clock). These are some useful commands:
acpi #Display battery life
date #Display time and date

For sound I installed gnome-sound-applet which launches into trayer.

Step 5. Confession: Ok, so while what Step 4 said is correct, I actually ended up installing gnome entirely to allow me access to all of the easy-to-use gnome apps that I'm so used to. I know there are better options but this way I was easily able to use gnome specific apps and tools from inside xmonad

Step 6. Suspend:
I made a script file to suspend the computer, and here is the command I use:
echo -n mem > /sys/power/state

Step 7. Enjoy! xmonad is a great way to maximize screen real-estate when you have to deal with small screens (not 4 monitors, but like a net-book). Have fun and good luck!

Sunday, January 29, 2012



Here is a poem I wrote in terza rima. Enjoy!

Linux


Linux is the king OS
its power unmatched
by any of the rest

built from an idea hatched
by the godly Linus Torvalds
whose ideas dispatched

out to all the many small
communities of developers around,
this idea of an OS, the best of all

Its birth was the crown
of a new era of immaculacy
to burn microsoft into the ground

microsoft, for even with its maturity
still feels the need to screw
its users by faltering in security

and with that I call upon you
to make the switch to Backtrack
or Debian or even Ubuntu

and fight the power, fight back!
together we will show microsoft
that we are ready to attack

Wednesday, January 11, 2012

Linux == God [Video]




I just finished watching "The Story of Linux: Commemorating 20 Years of the Linux Operating System" (Yes, I should have watched this sooner.)
-1:18 for Linux Jesus Drawing

Where I learned of IBM Linux Ads - 2:14

Which led me to this: IBM Linux ad - Prodigy

Saturday, January 7, 2012

Start My Day - Linux Script For Daily Apps


Click. Click. Click. Every day when I sit down at my computer and boot it up, I always launch the same 6 applications. I got tired of opening them manually every time, so instead I put a shortcut on my desktop that allows me to launch all of my apps at once via bash script.

To do this, just make a file called <whatever>.sh and put it somewhere safe. Next, create a desktop shortcut (Gnome 2) or menu entity (Openbox) that points to that script file. Then just add whatever commands you want to run to <whatever>.sh and you're good to go.

About Shell Scripting:
I recommend (if you're not familiar with it) reading more into shell scripting and really learning it, as it is a great tool for many jobs with this one being a very small example.

My script:
I keep my stuff simple, <command> &\n, nothing fancy.
google-chrome &
firefox http://gmail.com &
pidgin &
etc...

Notice the "&" at the end of each command. This tells the interpreter to run the process in the background, and continue running the next command.
The URL after firefox tells it to open that url, just how it would work in a terminal shell.

Autorun
Now, if you want your script to run automatically at login then the easiest option is to use the .bashrc_login file. At login time, bash looks for this file in your home directory and runs it. It should be located under your home directory ~/.bashrc_login. If it doesn't exist just create a new one. Add the command "sh /path/to/file/<whatever>.sh &" to the end of the file and you're set.

Further Reading
Running multiple apps at once is great, but bash scripts can do so much more. Here is a great guide to let you unleash your inner ninja: http://tldp.org/LDP/abs/html/