Setting up bup as my backup system, making it act like a timemachine.

This is one of those: 'writing this post to make sure I remember how I've done this'.

+William Stein posted about bup which he is using to backup +The Sagemath Cloud (if you haven't seen that before make sure you go check it out, here's a video in which I describe it: http://goo.gl/5DtYQq).

bup is a piece of backup software based on git. Here's a talk by +Zoran Zaric explaining it:

The documentation isn't too great for bup, this is the blog post I found the most helpful on it: http://kacper.blog.redpill-linpro.com/archives/tag/bup

the ubuntu man pages are also pretty helpful.

Anyway, here's how I setup bup to work like apple's time machine.

Once bup is installed (super easy following readme instruction on Mac OSX and ubuntu). I run:

$ bup -d pathtochosenharddrive init

By default bup uses the ~/.bup directory for everthing. Using the -d flag tells bup to run whatever command (in the above instance: init)  in a chosen hard drive. If you're happy to backup to your ~ then ignore all instances of -d pathtochosenharddrive in the following. (Note you can also change $BUP_DIR to take care of this, and you'll also need to know the path to your given hard drive).

This initialises a git repository (you only need to do this once really).

I put the following in a script (backup.sh):

bup -d pathtochosenharddrive index -ux /directorytobackup
bup -d pathtochosenharddrive save -n backupname /directorytobackup

The first line indexes the files (the -ux flags are something to do with recursively going through the files: type man bup index to read more). The second line checks the index and then saves all files as required (giving them a name).

To setup this backup script to run every hour I write the following to a txt file (crontab.txt):

0 */1 * * * globalpathtobackupscript/backup.sh

To add this to the cron jobs:

$ crontab crontab.txt

If you type:

$ crontab -l

You should see the the contents of the crontab.txt file now added to the scheduled jobs. The first 0 implies that it'll run at the 0th minute, the */1 means every one hour (so you can easily change this), the other * mean 'every', day, month and day of the week.

The first time you run this it should take a fair while (especially if you're backing up your whole ~) but afterwards it shouldn't take too long at all.

To check what bup has done, run:

$ bup -d pathtochosenharddrive ls

That should return:

backupname/

and/or any other names of backups. If you want to see the actual backup snapshots:

$ bup -d pathtochosenharddrive ls backupname

which will return a list of timestamped snapshots.

This has been working pretty seamlessly for a week for me now and I'm probably going to set it up on my work Mac instead of timemachine.

Setting up Zotero to use Dropbox for your attachments

So I've used +Mendeley in the past but recently started using Zotero to handle my references. There are certain aspects that I find +Mendeley a bit more user friendly for but having changed my workflow slightly I'm now a huge fan of Zotero (the scrapping tool is far superior to +Mendeley's ).

I don't use the firefox version but the standalone app which runs great on my linux box and my Mac.

I have a bunch of Dropbox space and the fact that Zotero doesn't play super nice with +Dropbox was a bit of a pain (you just had to point Mendeley at your +Dropbox folder and you were done). For a while I've just been using Zotero's base online storage but today I've just set it up to work on Dropbox so I thought I'd record how I've done it.

I asked a while back about this on +Google+ and had a bunch of people telling me: oh just use symbolic links it's super easy. Symbolic links have been one of those things I've been meaning to understand for a while but at that moment in time I just nodded and smiled. This morning I've just taken the time to figure it out and in particular see how to set it up to work with Zotero. It is super easy and so I thought I'd write a post in case it helps anyone and also to make sure I remember how to do it. I also didn't seem to be able to find anywhere online that explains the second half of what you need to do so that's here too.

First of all, if you check forums and stuff, there's a big safety warning with using Dropbox with Zotero. I understand that this is mainly because Zotero has multiple things going on: a database (that knows what is what and who wrote what etc) but also a storage folder (which contains the pdfs). This post is about division of labour: we're going to setup Zotero to take care of the database and +Dropbox to take care of the pdfs. There are basically 2 steps on 1 computer and 2 on every other.

If you use the preferences and change the data directory to dropbox, then if you're running on more than one box you will most likely corrupt the database (which is a bad thing: 'run you fools!').

So basically: don't mess with your data directory.

The 'solution' (I think) is to use 'symbolic links' to trick your computer in to thinking that your storage folder is also on your +Dropbox ( +Dropbox are the 'idiots' here and won't know the difference and just sync it all).

So to do that, choose computer 1. On computer 1 you setup a symbolic link from your Zotero storage file to your +Dropbox folder. This is what I did:

Step A1. Go to your preferences in Zotero and click on 'Show Data Directory'. This will open up your zotero folder (which contains the storage folder that we're looking for). Remember the path for this (click on info, or properties or something).

Step A2. Now for the magic trick: we create a symbolic link:

ln -s paththatitoldyoutoremember/storage ~/Dropbox/Literature/ZoteroStorage/

This tells computer 1 (and +Dropbox) that there's a folder in /Dropbox/Literature/ that contains folders with all your pdfs (it actually contains directories for each file):

(There is however no such folder, just a symbolic link that tricks everyone involved in to thinking that there is such a folder.)

That is however not everything. You now need to go to your other machines and tell Zotero on there that the storage file isn't exacly what it thinks it is (we trick it).

Step B1. So on computer 2 (and any other computer) go to the Zotero folder (remember just click on 'Show Data Directory' to find the path which you want to remember ie copy to your clipboard) and delete the storage folder (I think: be careful, don't sue me...):

rm -r otherpaththatitoldyoutoremember/storage

(make sure it deletes, when I did this on one machine I had to get rid of the folder again for some reason)

Step B2. Once we've done that we need to tell zotero not to worry and create a symbolic link of the storage folder (which it thinks is an actual folder) that is now in our +Dropbox:

ln -s ~/Dropbox/Literature/ZoteroStorage/ otherpaththatitoldyoutoremember/storage

That's basically it. If you now take a look at that folder/symlink on computer 2 you'll see all the folders (containing the pdfs) from your other machine (I'm blurring a bit of the path in case that somehow tells you my credit card number):

Now if you add a new file to Zotero and any given computer +Dropbox will first of all copy over the pdfs to all the right places (the symbolic links take care of the pdfs) and then when you sync zotero it'll also have the correct data (Zotero takes care of the database).

(Finally you can go to your preferences on Zotero on all your machines and turn off attachment syncing as +Dropbox is now taking care of that).

Handling data files in a Sage notebook (and some linear regression in Sage)

One of my most viewed videos on +YouTube is the following (briefly demonstrating how to import csv files in to +python):

I'm in the middle of preparing various teaching materials for an upcoming class on coding for mathematicians. I'm teaching this class in a flipped classroom (if you don't know what that is circle +Robert Talbert on G+ who posts a lot of great stuff about it) and as a result I've been screen casting a lot recently. Some of these clips are solely intended for my students (as I don't believe they'd be of interest to anyone else 'to do this exercise try and think about what your base case would be for the recursion to terminate'). I'm just starting to screen cast for the +Sage Mathematical Software System part of the course and needed to put together a little something as to how to import data in to a Sage notebook. As the above video seemed quite helpful to people I thought I'd put together another one that might be helpful.

The data file used can be found here.

Here are the lines of code used in the notebook:

import csv

f = open(DATA + 'reg', 'r')  # Open the data file using the special DATA variable
data = csv.reader(f)
data = [row for row in data]  # Read data using csv library
f.close()

data = [[row[2], row[1]] for row in data[1:]]  # Only use data that is of interest, remove unwanted columns and 1st row

a, b = var('a, b')  # Declare symbolic variables
model(t) = a * t + b  # Define model

fit = find_fit(data, model, solution_dict=True)  # Find fit of model to data

model.subs(fit)  # View the model

p = plot(model.subs(fit), 5, 11, color='red')  # Plot fit
p += list_plot(data)  # Plot data
p

Creating a slideshow video with YouTube in 4 + 1 ridiculously easy steps.

I've been taking photos on my way to work quite regularly of the construction of a new building by Cardiff University: http://goo.gl/k9qYN. I've been posting them on G+ which has been nice as you can click through and see all the photos and click through to see the progression of the building: http://goo.gl/CnFGd1.

However, now that I've stopped taking the pictures (the building is open) I thought I'd try and put together a video slideshow. 

My immediate thought was to see if the YouTube video editor would allow me to do this: it does and it's really easy!

1. Click on 'Upload' on YouTube which brings up the screen below. From there click on 'Photo slideshow':

2. Either upload photos (I guess) or go straight to your G+ albums. You can see in the screenshot that my 'Cardiff University Maindy Park Construction' is right there:

3. The photos come up right there and you can change the order if need be:

4. You can then mess with some settings and include some music:

5. That's pretty much it but you can click on 'Advanced editor' which brings up the usual video editor that you can use to add annotations and stuff to the video.

Here's the video I put together:

I really like how all these tools talk to each other. As the photos I take go straight to my G+ account anyway through instant backup I might well do a couple more of these which I could always make 'unlisted' if I don't think they're of public interest so that it's easy to share with friends and family.