Showing posts with label regression. Show all posts
Showing posts with label regression. Show all posts

Friday, 11 October 2013

Revisiting the relationship between word counts and code word counts in LaTeX documents

In this previous post I posted some python code that would recursively search though all directories in a directory and find all .tex files. Using texcount and wc the code the script would return a scatter plot of the number of words against the number of code words with a regression line fitted.

Here's the plot from all the .tex files on my machine:



That post got quite a few views and +Robert Jacobson was kind enough to not only fix and run the script on his machine but also sent over his data. I subsequently tweaked the code slightly so that it also returns a histogram. So here's some more graphs:

  • Robert's teaching tex files:




  • Robert's research files:



It looks like my .6 ratio between code words and words isn't quite the same for Robert...

BUT if we combine all our files together we get:



So I'm still sticking to the rule of thumb for words in a LaTeX file: multiply your number of code words by .65 to get in the right ball park. (But more data would be cool so please do run the script on your files :)).

The tweaked code (including Robert's debugging) can be found in this github repo: https://github.com/drvinceknight/LaTeXFilesWordCount

Saturday, 5 October 2013

Almost a 2 to 1 ratio of total code words to words in my LaTeX files...

In my previous post I posted a small python script that will recursively go through all directories in a directory and return the word count distribution using texcount (a utility that strips away LaTeX code to count words in documents). In this one I'm going to try and find a way of finding out how many words are in my LaTeX files without counting them (kind of).

On G+ +Dima Pasechnik suggested the use of wc as a proxy but wc gives the count of all words (include code words). I thought I'd see how far off wc would be. So I modified the python script from my last post so that it not only runs texcount but also wc and carries out a simple linear regression (using the stats package from scipy). The script is at the bottom of this blog post.

Here's the scatter plot and linear fit for all the LaTeX files on my system:




We see that the line $y=.68x-27.01$ can be accepted as a predictor for the number of words in a LaTeX document as a function of the total number of code words.

As in my previous post I obviously have an outlier there so here's the scatter plot and linear fit when I remove that one larger file:

The coefficient is again very similar  $y=.64x+26$.

So based on this I'd say that multiplying the number of codewords in a .tex file by .6 is going to give me a good indication of how many words I have in total.

Here's a csv file with my data, I'd love to know if other people have similar fits.

Here's the code (a +Dropbox link is here):

#!/usr/bin/env python import fnmatch import os import subprocess import argparse import pickle import csv from matplotlib import pyplot as plt from scipy import stats parser = argparse.ArgumentParser(description="A simple script to find word counts of all tex files in all subdirectories of a target directory.") parser.add_argument("directory", help="the directory you would like to search") parser.add_argument("-t", "--trim", help="trim data percentage", default=0) args = parser.parse_args() directory = args.directory p = float(args.trim) matches = [] for root, dirnames, filenames in os.walk(directory): for filename in fnmatch.filter(filenames, '*.tex'): matches.append(os.path.join(root, filename)) wordcounts = {} codewordcounts = {} fails = {} for f in matches: print "-" * 30 print f process = subprocess.Popen(['texcount', '-1', f],stdout=subprocess.PIPE) out, err = process.communicate() try: wordcounts[f] = eval(out.split()[0]) print "\t has %s words." % wordcounts[f] except: print "\t Couldn't count..." fails[f] = err process = subprocess.Popen(['wc', '-w', f],stdout=subprocess.PIPE) out, err = process.communicate() try: codewordcounts[f] = eval(out.split()[0]) print "\t has %s code words." % codewordcounts[f] except: print "\t Couldn't count..." fails[f] = err pickle.dump(wordcounts, open('latexwordcountin%s.pickle' % directory.replace("/", "-"), "w")) pickle.dump(codewordcounts, open('latexcodewordcountin%s.pickle' % directory.replace("/", "-"), "w")) x = [codewordcounts[e] for e in wordcounts] y = [wordcounts[e] for e in wordcounts] slope, intercept, r_value, p_value, std_err = stats.linregress(x,y) plt.figure() plt.scatter(x, y, color='black') plt.plot(x, [slope * i + intercept for i in x], lw=2, label='$y = %.2fx + %.2f$ ($p=%.2f$)' % (slope, intercept, p_value)) plt.xlabel("Code words") plt.ylabel("Words") plt.xlim([0, plt.xlim()[1]]) plt.ylim([0, plt.ylim()[1]]) plt.legend() plt.savefig('wordsvcodewords.png') data = zip(x,y) f = open('wordsvcodewords.csv', 'w') wrtr = csv.writer(f) for row in data: wrtr.writerow(row) f.close()

Monday, 9 September 2013

Handling data files in a Sage notebook (and some linear regression in Sage)

One of my most viewed videos on +YouTube is the following (briefly demonstrating how to import csv files in to +python):



I'm in the middle of preparing various teaching materials for an upcoming class on coding for mathematicians. I'm teaching this class in a flipped classroom (if you don't know what that is circle +Robert Talbert on G+ who posts a lot of great stuff about it) and as a result I've been screen casting a lot recently. Some of these clips are solely intended for my students (as I don't believe they'd be of interest to anyone else 'to do this exercise try and think about what your base case would be for the recursion to terminate'). I'm just starting to screen cast for the +Sage Mathematical Software System part of the course and needed to put together a little something as to how to import data in to a Sage notebook. As the above video seemed quite helpful to people I thought I'd put together another one that might be helpful.



The data file used can be found here.

Here are the lines of code used in the notebook:

import csv

f = open(DATA + 'reg', 'r')  # Open the data file using the special DATA variable
data = csv.reader(f)
data = [row for row in data]  # Read data using csv library
f.close()

data = [[row[2], row[1]] for row in data[1:]]  # Only use data that is of interest, remove unwanted columns and 1st row

a, b = var('a, b')  # Declare symbolic variables
model(t) = a * t + b  # Define model

fit = find_fit(data, model, solution_dict=True)  # Find fit of model to data

model.subs(fit)  # View the model

p = plot(model.subs(fit), 5, 11, color='red')  # Plot fit
p += list_plot(data)  # Plot data
p

Tuesday, 2 April 2013

Handling dates and carrying out linear regression to model number of members of a Google Plus community in Python

I've blogged about rugby before (using some game theory to disagree with a rugby commentator) but here's another one.

I'm a member of the Rugby Union community on Google plus and it's a great place to chat about rugby and share stuff. As I said on this post about G+, I don't post publicly about rugby as I assume that most people who circle me won't be too interested in my opinions on rugby so it's nice to have a place to go. Anyway I really recommend the community it's a nice place if you like the game.

+Davide Coppola who's the owner of the community has been posting an announcement every time we go up a 100 members. We're currently at about 960 people and we've been wondering when we'll go past the 1000 member mark.

I thought I'd code a very simple bit of python to have an educated guess but also to see what member numbers have looked like.

Here's the data:

Date,Number
2013-04-02,961
2013-03-26,900
2013-03-13,800
2013-03-03,700
2013-02-19,600
2013-02-10,500
2013-02-08,400
2013-01-31,300
2013-01-06,200
2012-12-17,100
2012-12-07,1

First of all I needed to import all the data (I've posted a short screencast about handling csv data in python and here's an old blog post with the code):

import csv
outfile = open("Data.csv", "rb")
data = csv.reader(outfile)
data = [row for row in data]
outfile.close()

Once that is done I use the datetime python module to convert the dates read as strings to actual dates but also to create a list containing the member numbers:

dates = [datetime.datetime.strptime(e[0], '%Y-%m-%d') for e in data[1:]]
numbers = [eval(e[1]) for e in data[1:]]

I will use the dates list later on to plot things nicely but for now I need to consider numeric data (to fit a linear model). I convert the dates list to a set of numbers counting the number of days having passed from the first day of the community:

x = [(e - min(dates)).days for e in dates] 

Once I've done that I use the stats sub package from the scipy library to carry out a simple linear regression:

from scipy import stats
gradient, intercept, r_value, p_value, std_err = stats.linregress(x, numbers)

(I in fact only need the gradient and intercept from the above but it's all there in case I wanted it.)
To find out when we can expect to go past 1000 members (assuming a linear model of growth):

projected_date = min(dates) + datetime.timedelta(days=(1000 - intercept) / gradient)

To project the linear fit to see what number of members we could hope to have after a year I do the following:

extra_date = min(dates) + datetime.timedelta(days=365)
projection = gradient * (extra_date - min(dates)).days + intercept

Finally to plot all of the above I use pyplot:

import matplotlib.pyplot as plt
plt.figure(1, figsize=(15, 6))
plt.plot_date(dates, numbers, label="Data")
plt.plot_date(dates + [extra_date], numbers + [projection], '-', label="Linear fit (%.2f join per day)" % gradient)
plt.legend(loc="upper left")
plt.grid(True)
plt.title("Rugby Union Google Plus Community Member Numbers")
plt.savefig('Rugby_Union_Community_Numbers.png')

The output is given here (there's something not great with the graph: I don't know why a line is being plotted between the data points but I haven't been able to fix that easily):



We should have about 3000 member by the end of 2013 and most importantly we can expect to go past the 1000 member number on the 9th of April 2013 at 21:38 (+Davide Coppola and +Andrew Byrne: set your alarm clocks).

Note that you would in fact want to fit a much more complex model than a simple linear fit to actually try and forecast any of what I've done above. I was in fact suprised at how linear the fit was...

It's been a quick bit of fun and perhaps could prove useful to some as to how to handle dates (and do a simple linear regression) in python.

I've actually got a couple of screencasts that show how to handle dates in R and SAS which I'll put here in case they're of use to anyone.