Reflecting On Evidence For The Benefits Of Active Learning

A paper entitled 'Active Learning Increases Student Performance in Science, Engineering and Mathematics' (by Freeman and 6 co-authors) was recently published and got a fair bit of quasi-popular press exposure as the message seemed to be quite simple: Students learn better when they are not simply an audience during contact time (or in other words: lectures aren't the best). I decided that this was a paper that I really wanted to read so I went through the pain of having to google twice (thus making it 'officially' hard to find) and have just finished reading it.

This is a great paper.

Firstly there is a nice definition of what active learning is:

"Active learning engages students in the process of learning through activities and/or discussion in class, as opposed to passively listening to an expert. It emphasizes higher-order thinking and often involves group work."

The definition given in the paper for a traditional lecture is given as:

"[...] student activity is assumed to be limited to taking notes and/or occasional and unprompted questions of the instructor." 

The definition for active learning is not obtained out of thin air but takes in to account over 300 audience members at departmental seminars on active learning throughout North America.

What I particularly like about the definition is that it specifically makes a point of mentioning the higher-order thinking.

I was recently lucky enough to attend a talk by Dr. Collin Jones who teaches in Tasmania using an active learning approach. Here is a diagram that I've put together trying (humbly) to get across what he said on the subject of active learning (you might want to open the image in a new tab...):

In that I'm trying to say that if we want students to access the higher levels of Bloom's taxonomy across the various topics in a course, with our help through contact time then a more active learning approach seems logical.

It looks like Freeman's paper confirms this.

It tells us lots of great stuff.

The first section quickly summarsies the findings of rigorously (as far as I can tell the rigour in this paper is great) analysing the results of 225 studies that obeyed the following criteria:

1. Involved a comparison between traditional lecturing and active learning;
2. Involved a regular class;
3. Only looked at comparison where the delivery method was the main change;
4. Involved a STEM subject;
5. Included an evaluation of academic performance

This later point is what has made this paper really stand up and get noticed. In fact this and 2 other papers (here and here) are the only papers that offer a meta analysis of student performances. This is not a paper reflecting on teaching methodologies but on the actual effects that teaching methodologies have on student learning.

There are a variety of well presented findings and I will try and summarise them very briefly here:

1. Students will perform by just under half a standard deviation better in an active learning environment;
2. Students are 1.5 times more likely to fail in a traditional lecture environment;
3. 1. and 2. are not affected by the particular STEM subject;
4. The gain in 1. for concept inventory style assessment are lower than for 'instructor-written' type assessment. This is argued (and I think I agree) to be in line with my arrows going through Bloom's taxonomy. In an active learning environment students are more likely to perform well on tasks that require a deeper learning;
5. The gain in 1. is more prominent in smaller classes. This also makes sense (taking a class with a minimum number of students of 1, it would be idiotic to lecture and similarly for an infinite class perhaps an active learning approach would not be appropriate).

All of the above is just plain cool to have a rigorous basis for. Potential weaknesses of the study are confidently rebutted in the manuscript itself (for example the number of papers showing no improvement whatsoever that would be needed to lower the impact of the work is shown to be high).

Evidence

+Stan Yoshinobu has written a brief post about Freeman's paper at The IBL blog and I really like:

"The perspective I like to take is one that is used in medicine. When new techniques or treatments are shown through evidence to provide better care for patients, then the medical community adopts those new practices."

This is something alluded to by Freeman et al. They talk about their paper (and others) being the 'first generation' of research in to active learning and that now that there is evidence for the benefits perhaps the second generation of research would start looking at best types of active learning for given students etc...

1 last thing.

The final thing I want to talk about is the potential weakness of this research. This is pointed out by the authors:

"The instructors who implemented active learning in these studies did so as volunteers. It is an open question whether student performance would increase as much if all faculty were required to implement active learning approaches."

My personal opinion is that that really hits the issue on the head. Instructors who innovate with their teaching so as to move away from the tried and tested lecture that has been in place for 100s of years are most probably the academics who have the energy and conviction to put a lot of effort in to their teaching.

Often in academia where the pressure to publish and obtain funding are immense it is sad to say that it's very easy to simply reach for the same set of notes that has always passed down and lecture. This leaves students to attempt to reach deeper levels of learning on their own.

It must also be said though, (as alluded to in the paper) active learning means a variety of things. For example I use problem based learning, flipped classrooms and a combination of active game playing (that I would like to think has some element of inquiry based learning) in my own teaching however active learning in Freeman's paper has a breadth of meanings which include the simple use of clickers to get some form of student engagement with the learning. So I don't necessarily think that an active learning approach implies a huge distancing from classic lecturing for all instructors (see the earlier definitions of the active learning and traditional lecture).

As the authors' state it's a case of now identifying the best pedagogic approaches for instructors as well as students: getting that pairing right.

Fun

I really enjoyed something that Dr Sue Rigby said at the opening of the 2014 HEA STEM conference:

"We have to teach in a way that is fun. We get to choose how we teach"

As educators I think it's very important to remember that we do indeed get to choose how we teach and ultimately active learning is often simply far more fun than traditional lectures (I sure had a smile on my face when trying to teaching cooperative game theory by having students play waste paper basketball).

We should always choose a pedagogic approach for a better reason than "that's how it has always been done". Sound research like the one in this paper seems like a good a reason as any.

-----

Edit: On the 7 and 8th of July (2014) +Paul Harper and I are hosting a workshop on active learning techniques. Please take a look :)

Wizards, Giants, Linear Programming and Sage

I've been playing a game on my phone for a couple of months now called Clash of clans. It's one of those freemium games that tells me to do something on my phone every now and then:

In essence during the game you build your base with buildings to obtain resources, create troops and defend against attacks.

Here's a screen shot of my base at the moment:

As well as building your base, you can also put together a raiding party and attacking other bases.
In this post, I'll describe the use of a mathematical technique called linear programming so as to get the best combination of Giants, Wizards etc in a raiding party.

To train a warrior costs Elixir (a resource that you mine and/or plunder) but also costs space in your army camps.

Here are the various warriors available to me at the moment (as you build better buildings you get more warriors etc...).

1. Barbarian (Damage per second: 18, Hitpoints: 78, Housing Space: 1, Elixir Cost: 80):

2. Archer (Damage per second: 16, Hitpoints: 33, Housing Space: 1, Elixir Cost: 160)

3. Goblin (Damage per second: 19, Hitpoints: 36, Housing Space: 1, Elixir Cost: 60)

4. Giant (Damage per second: 24, Hitpoints: 520, Housing Space: 5, Elixir Cost: 2000) 

 5. Wall Breaker (Damage per second: 32, Hitpoints: 35, Housing Space: 2, Elixir Cost: 2500)

6. Balloon (Damage per second: 72, Hitpoints: 280, Housing Space: 5, Elixir Cost: 3500)

7. Wizards (Damage per second: 125, Hitpoints: 130, Housing Space: 4, Elixir Cost: 3000)

8. Healer (Damage per second: NA, Hitpoints: 600, Housing Space: 14, Elixir Cost: 6000)

9. Dragon (Damage per second: 140, Hitpoints: 1900, Housing Space: 20, Elixir Cost: 25000)

To summarise the information about the players I'm going to put the costs in a vector \(c\) and the housing space in a vector \(h\) (indexed using the same order as the images above):

\[c=(80,160,60,2000,2500,3500,3000,6000,25000)\]
\[h=(1,1,1,5,2,5,4,14,20)\]

Currently I have a capacity of 50 housing spaces per camp and I have 4 camps so I have an upper bound on the total amount of housing space of \(K=200\).

If I let the vector \(x\) denote a raiding party so that \(x_i\) is the number of warriors of type \(\i) (using the ordering above) then we have the corresponding housing space constraint:

\[\sum_{i=1}^9h_ix_i\leq K\]

I can now try and do two things:

1. Get a full set of camps by minimising the overall cost.
2. Get the most powerful raiding party that can fit in my camps.

I will solve both of these problems using what is called Integer Linear Programming. The integer part of that is due to the fact that I must have whole parts of warriors.

1. Minimising my costs

The cost of a given rading party \(x\) is given by:

\[C(x) = \sum_{i=1}^9c_ix_i \]

Thus to minimise this cost and fill the camps we need to solve the following minimisation problem:

\[\text{minimise }C(x) = \sum_{i=1}^9c_ix_i\]

such that:

\[H(x)=\sum_{i=1}^9h_ix_i= K\]
and
\[x_i\in\mathbb{Z}_{\geq0}\]

How do we solve this? I've blogged about a similar type of optimisation problem that I use to schedule my class assessment and for that I used the awesome sagemath. You can read that blog post here. Here I will do the same thing.

The following Sage code is what is needed to create the linear program and also obtain the solution:

# Set up parameters:
K = 200  # Set the upper bound
c = [80,160,60,2000,2500,3500,3000,6000,25000] # The the cost vector
h = [1,1,1,5,2,5,4,14,20]  # Set the 'housing space' constraints vector
n = len(c)  # Get the number of warriors

C = lambda x: sum([x[i]*c[i] for i in range(n)])  # Define the cost function
H = lambda x: sum([x[i]*h[i] for i in range(n)])  # Define the housing capacity function

# Create and solve the optimisation problem
p = MixedIntegerLinearProgram(maximization=False)  # Create an optimisation problem with minimization as the goal
x = p.new_variable(integer=True)  # Create a variable with the requirement that it must be integer
p.add_constraint(H(x)==K)  # Set the housing constranit
p.set_objective(C(x))  # Set the objective function
p.show() # Show the optimisation problem
p.solve()  # Solve the optimisation problem

# Solve the optimisation problem

print 'Has solution:' 
for i, v in p.get_values(x).iteritems(): 
    print 'x_%s = %s' % (i, int(round(v))) 
print 'Housing spaces used: %s' % H(p.get_values(x))

The output is (disappointing but expected): \(x=(0,0,200,0,0,0,0,0)\) which implies that I should have 200 Goblins.

If you're familiar with the game this is not a very sensible way to go as Goblins are really only good at getting loot from defenceless bases.

We can add a couple more constraints so that we don't have too many goblins (let's limit it to 10) and also so that we have at least one healer:

p = MixedIntegerLinearProgram(maximization=False)  # Create an optimisation problem with minimization as the goal
x = p.new_variable(integer=True)  # Create a variable with the requirement that it must be integer
p.add_constraint(H(x)==K)
p.add_constraint(x[2]<=10)  # Don't want more than 10 goblins
p.add_constraint(x[7]>=1)  # Want 1 healer at least
p.set_objective(C(x))
p.show()
p.solve()
print 'Has solution:'
for i, v in p.get_values(x).iteritems():
    print 'x_%s = %s' % (i, int(round(v)))
print 'Housing spaces used: %s' % H(p.get_values(x))

This gives us \(x=(176,0,10,0,0,0,0,1,0)\).

This is again not very satisfying as we're basically filling our raiding party with the cheapest options. That was the purpose of this approach though. 

The next thing we'll look at is how to actually get a good raiding part.

2. Getting the 'best' raiding party.

As you can see on the images, every warrior is different: some fly, some are better against buildings, some have more hitpoints than others etc...

One way of getting the 'best' raiding party could be to maximise the damage per second of the raiding party. To do this, let's define the following vector (which is just the damage per second that can be read off of the images above):

\[d=(18, 16, 19, 24, 32, 72, 125, 0, 140)\]

Now our maximisation problem can be written as:

\[\text{maximise } \sum_{i=1}^9d_ix_i\]

such that:

\[H(x)=\sum_{i=1}^9h_ix_i\leq K\]
and
\[x_i\in\mathbb{Z}_{\geq0}\]

Here's the Sage code to do this:

damagepersecond = [18, 16, 19, 24, 32, 72, 125, 0, 140]
C = lambda x: sum([x[i]*damagepersecond[i] for i in range(n)])

p = MixedIntegerLinearProgram()  # Create an optimisation problem with minimization as the goal
x = p.new_variable(integer=True)  # Create a variable with the requirement that it must be integer
p.add_constraint(H(x)<=K)
p.add_constraint(x[2]<=10)  # Don't want more than 10 goblins
p.add_constraint(x[7]>=1)  # Want 1 healer at least
p.set_objective(C(x))
p.show()
p.solve()
print 'Has solution:'
for i, v in p.get_values(x).iteritems():
    print 'x_%s = %s' % (i, int(round(v)))
    print 'Housing spaces used: %s' % H(p.get_values(x))

The result of the above is \(x=(0,0,2,0,0,0,46,1,0)\). I.e take 2 goblins, 46 wizards and 1 healer.
The problem with this raiding party is that wizards are actually quite fragile (archers towers and cannons can pretty much just pick them off).

So perhaps instead of maximising the damage per second we could maximise the total hit points of the party.

To do this we need to define the vector of hitpoints:

\[p=(78, 33, 36, 520, 35, 280, 130, 600, 1900)\]

Now our maximisation problem can be written as:

\[\text{maximise } \sum_{i=1}^9p_ix_i\]

such that:

\[H(x)=\sum_{i=1}^9h_ix_i\leq K\]
and
\[x_i\in\mathbb{Z}_{\geq0}\]

This gives us \(x=(0,0,0,0,93,0,0,1,0)\): a raiding part of wall breakers.

The problem with this is that wall breakers are good for just that: breaking walls. They carry a bomb up to a wall and then they explode. So let's add some more constraints to our optimisation problem so as to get something a bit more useful.

• Let's make sure that we always have at least 20 housing capacity worth of distance attackers (Archers or Wizards):

\[x_1h_1+x_6h_6 \geq 20 \]

• Let's ensure that we have at least 1 air attacker (so that once the air defenses are out of the way we can finish off the base):

\[x_5+x_8\geq1\]

• Let's ensure that we always have some wall breakers to clear the walls for our ground units (1 per 5 giants and 1 per 20 barbarians say):

\[5x_4\geq x_3\]

\[20x_4\geq x_0\]

The code to solve this is given here:

p = MixedIntegerLinearProgram()  # Create an optimisation problem with minimization as the goal
x = p.new_variable(integer=True)  # Create a variable with the requirement that it must be integer
p.add_constraint(H(x) <= K)
p.add_constraint(x[2] <= 10)  # Don't want more than 10 goblins
p.add_constraint(x[7] >= 1)  # Want 1 healer at least
p.add_constraint(x[1] * h[1] + x[6] * h[6]>=20)  # Want at least 20 capacity worth of distance attackers
p.add_constraint(x[5] + x[8]>=1)  # Want at least 1 of air attackers
p.add_constraint(5 * x[4] >= x[3])  # Want at least 1 wall breaker per 5 giants
p.add_constraint(20 * x[4] >= x[0])  # Want at least 1 wall breaker per 20 barbarians
p.set_objective(C(x))
p.show()
p.solve()
print 'Has solution:'
for i, v in p.get_values(x).iteritems():
    print 'x_%s = %s' % (i, int(round(v)))
print 'Housing spaces used: %s' % H(p.get_values(x))

This gives \(x=(1,20,0,23,5,0,0,1,2)\): so take 1 Barbarian, 20 Archers, 23 Giants, 5 Wall Breakers, 1 Healer and 2 Dragons.

The above would give us the most hit points but now we're missing out on the damage power of wizards.

Perhaps we could try and balance both hitpoints damage.

In essence we have what's called a multi objective optimisation problem. One approach to solve this is to simply weight both objectives.

We're going to use \(\alpha\) to denote the importance of the hitpoints and \(1-\alpha\) the importance of the damage.

Our optimisation function then becomes:

\[\alpha\sum_{i=1}^9p_ix_i+(1-\alpha)\sum_{i=1}^9d_ix_i\]

Using the same constraints as before and taking \(\alpha=.5\) (so that both damage and hit points have equal weighting) we get \(x=(11,0,0,25,5,0,5,1,1)\). Thus we should be taking in to battle: 11 Barbarians, 25 Giants, 5 Wall Breakers, 5 Wizards, 1 Healer and 1 Dragon.

This feels much more balanced than anything we've had so far and not far off what I've been using as my own raiding party.

Given that everything is nicely coded we can quickly obtain the make up of the optimal raiding party for a range of values of \(\alpha\):

We see that as \(\alpha\) increases (so that we care more about hit points) the number of Wizards we should use slowly decreases (at one point the constraint for ranged attacking units kicks in and we replace the Wizards with Archers).

You could easily tweak the constraints to perhaps ensure your party always has some Goblins or otherwise. I think this is a nice example where a mathematical model can be used to solve a problem but at the same time must not be taken far from the problem itself. There might be some players who only choose to attack bases with a particular type of defence so they can tweak/add constraints to the problem accordingly.

Feel free to play around with the code in the Sage cell below:

The mathematical techniques used in this post rely on Linear Algebra, Higher Dimensional Geometry and obviously the ability to write code :)

(On another note I'm still not in a clan on the game so if anyone has any space in a clan let me know!)

Debugging Python with pdb at the command line

I've just put up a short video (7 mins) describing one way of debugging in Python using the pdb module (here's the docs):

In this video I show how to step through code, view values of variables and also insert various types of break points.

All this is done interactively at the command line but you can also use the pdb library to include debugging commands in your Python script itself.

Reflecting on Pechu Kucha

In this previous blog post I described my frustration with the fact that the Pechu Kucha presentation style was imposed on all speakers at the #HEASTEM14 conference.

That was 3 days ago and with the conference behind me, I am going to write down my thoughts.
First of all: the whole conference was awesome. I had a great time and learnt a lot through the awesome discussions that arose after a session of talks. Each session involved a triplet of Pechu Kuchas (with no gaps) followed by a lengthy discussion.

These discussion were awesome.

Sadly I don't believe these discussions had anything to do what so ever with Pechu Kuchas.

I actually enjoyed my own Pechu Kucha but didn't really feel that it was anything different to a normal presentation (apart from no interaction with the audience and that it was shorter than usual).
I overheard a few speakers say things like: 'it really makes you rehearse' but personally rehearsing is something I always do (I probably rehearsed less for this as I was only talking for 3 minutes and 20 seconds).

The reason the discussions were so awesome was simply that there was enough time for them. The reason there was enough time for them was that Pechu Kucha ensured that everyone spoke for no more than 7 minutes.

This could have been achieved by ensuring that sessions were chaired strictly without the need to further constrain the speakers.

One comment on my previous post said: 'Talks overrun because moderators are not doing their job'.

I realise that it's not always easy to be a chair but I think that imposing Pechu Kucha is not the right solution.

One private comment on my previous post said that 'I also think that this format is not helpful for speakers with slight speech impediments'. Indeed, forcing everyone to use a given delivery style is not terribly inclusive.

I mentioned my opinion to a couple of organisers (again: who did a truly awesome job with the conference) and after clarifying that I wasn't worried about my personal experience of Pechu Kucha but more that I didn't think it was great that everyone had to use it, I received responses along the lines of:

- 'Well I suppose people knew that this was the way so they could choose to not give a presentation';
- 'There are other options such as presenting a poster'.

I perhaps misunderstood and/or am taking that out of context (in which case I apologise) but at an education conference attended by people for whom inclusivity is a high priority I can't say I was terribly satisfied with that response.

Finally to return to the great thing about this conference: the post talk discussions. These were truly awesome and it was so great to be in a place where everyone cared and these discussions offered a great opportunity to transfer ideas and opinions.

Sadly (as is to be expected) having a single discussion session at the end of 3 talks often meant that 1 or 2 of the talks received no discussion what so ever. This could of course be because the talk in question did not instigate enough interest for any questions but I also think that it could be because after the first question is asked and answered, further discussion just snowballed it's way through the room (which is awesome). This is a minor point for me, as in a way if the discussion didn't cover a particular talk but was very interesting perhaps it's not a problem (people could always approach the speakers after the talk).

To summarise:

- It was fun to try out the Pechu Kucha style.
- It worked (indeed there is no need for long talks, short talks are awesome)
- I don't think it should be enforced (this is NOT inclusive, what if I did not want to use PowerPoint? 'Not presenting' or 'presenting a poster' are not acceptable alternatives)

I would (humbly) suggest that next time Pechu Kucha is not imposed (perhaps a Pechu Kucha session or 2 would be a good idea) but I certainly think having a 7 minute limit on talks is a GREAT idea.

This would require strict chairs that kept the conference to time. Something similar to the 'Grand Council' I used in my first year class this year. It involved 40 1 minute pitches from my students: they each had a minute and I was quite brutal in my chairing of the session. I had a timer go off and would yell "YOU'RE DONE!" but mostly it was the next student talking that would usher the previous off the stage. In practice I think I actually only cut 5 students short as all the others kept to time brilliantly. +Paul Harper wrote this post about it.

I loved the conference. 

Pechu Kucha certainly didn't make it worse than it would have been otherwise as everyone made the system work.

I just don't feel that it was an inclusive way of doing things (nor that it was necessary). Next year I will hopefully attend and either use one of the Pechu Kucha workarounds (discussed in the comments on my previous post), present using the delivery system of my choice or I suppose: present a poster.

Finally similar to my image of Machu Picchu in my other post here is a picture of Pickachu (which is what I have been calling Pechu Kucha for the past 3 days):