jon pipitone

On leave

2011-07-19T18:54:00.006-04:00

Most of you that read this blog know this already: I've taken a leave of absence from the PhD program. I'd like to reflect on this move. When I mention that I've taken leave to anyone, the two questions I often get are, why? and what are you going to do?

So, first the why. I've been struggling with my life in academia since I began the master's program in 2007. The story I tell myself is that I've found it a real challenge to find satisfaction in the pursuit of computer science research. Mostly it has seemed at odds with all of my other interests in the environment, food, mental health, and activism. Of course, my environmental interests have strongly guided all of the research work I've done so far; my thesis work on software quality wasn't just about any old software, it was specifically about climate software for the express purpose being of some benefit to climate scientists. I've also run or participated in workshops related to climate change for highschoolers and for software researchers, or on software research with climate modellers, participated in many many discussions in our lab exploring the application of software research to climate change and generally to transition. Starting back in August of last year, I took a step back from this pursuit and wrote a post acknowledging my interest in psychology and environmentalism, and later about my need to incorporate activism into my research. These recognitions of my broader interests (beyond just the science of climate change) led me through a year with some wonderful courses and activities.

To essentialise, my recent academic focus has been entirely on exploring the question of what software research can contribute in the transition to a sustainable society. I've been trying to find a way to integrate my interests. At points I've been truly hard on myself for not having a good answer to this question, or for not even enjoying the exploration of it. In the time I've spent exploring this question (a little over three years now), I've not found anything that has strongly held my interest. I've found nothing in the mix of computer science and environmentalism that inspires me. Moreover, I feel as if I've exhausted myself in the effort of trying to find it.

I'm pretty sure my exhaustion is as a result of the way I've been approaching research (that is to say, it's not that I think the project of computational sustainability is intellectually bankrupt). I wish I could tell you in detail about what my approach has been and why I think it has led to this exhaustion (beyond being hard on myself), but I'm only now starting to get the distance to be able to reflect with any clarity. At the moment, it's still all a bit of a muddle.

I spent a long while considering the move to take a leave of absence. Originally when I transitioned into the PhD program I had set myself an arbitrary period of a year to discover a research project I felt was worth pursing. As my deadline approached, I started to feel a real anxiety over the decision. It felt uncomfortably binary: stay and keep searching, or give up the search and leave the program.

Staying began to feel like condemning myself forever to a wandering purgatory, and leaving felt like giving up an amazing privileged and opportunity. I had myself thoroughly wedged between these two options. To add to this drama, I can be a moody person and with some regularity find myself in multi-week long slumps (which vary from acedia to a more deep and despairing depression). Stir this all together and I was feeling pretty darn incapable of much other than just going through the motions.

In the depths of this state I spoke with one of my friends and mentors, who pointed out that I seemed to be obsessing with doing. As in, all I was asking myself were questions about what I should do next, and what I should be doing in order to be of benefit, or do in order to get myself out of my muddle, etc. He noted that I seemed to be neglecting being, and that maybe it was important for me that I actually do not try to make a decision to do anything. To resist making a decision would allow me the space to get a better sense of myself and my interests. And so, I took the option of a leave of absence. In effect, it's a decision to not make a decision.

That's the why.

This leave isn't just a delay though, a pause button. The important thing for me is how I am conducting myself now that I've left, and the attitude I bring to this time. I've been earnestly trying to explore this idea of being rather than doing. I'm typically always compelled to do things, to be productive in some sense, to take some action in the world that has some impact. But right now I am attempting to take a step back from myself and observe what's happening when I have those impulses. What is driving them? In what ways am I attached to them? What happens when they are frustrated, or when I purposefully choose an indeterminate path that neither satisfies or obstructs? What would happen if they were never satisfied, then who would I be? The attitude I'm trying to take with all of this is one of curiousity with myself, and an openness to not hold myself to the ways I, and others, have traditionally held me to.

If you must know, in terms of what I've actually been doing with my time: I have been journaling quite a bit, attempting to keep a regular habit of sleep and meditation, running much more, taking formal instruction in yoga, and a doing few other sorts of the known healthy things to incorporate into one's life. I've also been taking time out to sleep in the shade of trees, and appreciate the summer weather, and spend time with friends. Additionally, I've also been volunteering at the usual places, as well as reading books and listening to lectures on cognitive science and mindfulness.

So that's that: the why and the what. Now you know.

EDIT: I forgot to include one important thought that has been with me through all of this. I feel incredibly fortunate to be able to be in the position to even have this decision about school to make. And even more, to have the luxury now to take this time off from work altogether (though I am picking up the odd day of work here and there). My father calls this a "necessary luxury" which I think is a wonderfully oxymoronic, yet somewhat truthful, phrase.

Peak Oil Blues

2011-05-31T16:17:00.000-04:00

I'd like to introduce you all to the Peak Oil Blues blog. The blog is mainly written by Kathy McMahon, a clinical psychologist in Massachusetts, and it focuses on the psychological aspects of coping with peak oil, climate change, and collapse. It's fascinating.

I've been following the blog for some time now but I only recently dipped into some of the "Best of" articles. There was a lot of activity on the site a few years back and this page tracks and organises the most important posts. I wish I had started in on them sooner! Many of the questions I've been asking myself about coping with large changes are addressed in these articles with real sensitivity, self-awareness and depth.

The crop planning problem for market gardens

2011-04-09T12:34:00.000-04:00

I'm so happy that the Journal of Computers and Electronics in Agriculture exists; it is a wonderful combination of three topics that interest me. The papers tend to be about sensing and identification of animal or plant features, modelling and planning of agricultural systems, robotics, and decision-support tools. The papers themselves aren't usually my cup of tea though: most of the papers are aimed at solving problems that are relevant only to large or industrial agricultural operations, or for very specialized applications. But much of the technology proposed could, theoretically, be adapted and used with the kinds of farming I am interested in: small-scale, mixed farms and market gardens. In any case, I've been following this journal for a while now and I find it to be a fascinating read. It's especially fun when I run across papers like this, or this.

Anyhow, in order to prepare for my final assignment in Climate Informatics, I spent Friday afternoon scanning through the last five years of the publication ostensibly looking for articles to do with crop planning. (But, I also took the opportunity to look for any other interesting and inspiring articles -- hence the exhaustive search). The assignment for the class involves writing a research proposal and I am considering writing one to investigate the computational problem of finding an optimal cropping plan and rotation for a market-garden (I'm inspired by some of the work done by Institute for Computational Sustainability on equally unlikely computer science topics) . The crop planning problem could turn out just to be an annoyingly complex optimization problem without any research value, but I thought I'd check to see what prior work exists before I dismiss it.

Sadly, crop planning isn't something written too much about it in this journal. Here are the two relevant papers I found (the second isn't actually from CEA, but I found it by following references):

R. Sarker and T. Ray, "An improved evolutionary algorithm for solving multi-objective crop planning models," Computers and Electronics in Agriculture, vol. 68, no. 2, pp. 191-199, Oct. 2009.

J. Bachinger and P. Zander, "ROTOR, a tool for generating and evaluating crop rotations for organic farming systems," European Journal of Agronomy, vol. 26, no. 2, pp. 130-143, Feb. 2007.

The Sarker and Ray paper gets close to what I'm looking for actually, and has a wealth of references I could follow up with. The Bachinger and Zander article is much more focused on organic agriculture as it emphasizes the time dimension in the problem definition (rotating where crops are planted in a field over time is done to avoid nutrient depletion and pest problems -- this is not as big of a problem when your are using fertilizer and spraying pesticides). At the very least it shows that the topic is research-worthy in some field -- though I'm not yet sure it's worthy of computer science research.

Side note: In doing this review I was reminded of what is probably the most interesting and potentially world-changing (IMHO) applications of computers and electronics in agriculture: robotic weeding. Think about it. If we were able to develop effective precision robotic weeding systems the non-organic growers could all but give up herbicides. That's huge. And since weed control is such a big job for organic growers, robotic weeding devices means organic growers could do their jobs much more efficiently, and with less tractor work and backache.

Modifying historical commits in git

2011-03-22T16:32:00.000-04:00

As part of some work organising for a top secret workshop (soon to be discussed), I was given a git clone of a repository for participants to analyse on the condition that the commit messages and authors are removed. Fortunately, there is a straight-forward way to modify git history destructively (not something you'd normally want to do), using the git filter-branch command:

git filter-branch -f --env-filter '
export GIT_AUTHOR_NAME="`echo 798h23l$GIT_AUTHOR_NAME | md5sum`"; 
export GIT_AUTHOR_EMAIL="`echo 2i3fsdf$GIT_AUTHOR_EMAIL | md5sum`"; 
export GIT_COMMITTER_EMAIL="`echo d0af09$GIT_COMMITTER_EMAIL | md5sum`"; 
export GIT_COMMITTER_NAME="`echo 0sdfk9$GIT_COMMITTER_NAME | md5sum`";' --msg-filter '' -- --all

This blanks out the commit message, and sets the author and committer email and name field to a salted and hashed value (so that authors can be compared across commits, but not explicitly known).

Specifically, the --env-filter flag takes a chunk of shell code that sets various environment variables used to reset the author and committer email and name for each commit. I've set each variable to the md5sum of itself plus a salt. The --msg-filter also takes shell code to modify the commit message, but I've just set it to the empty string. The -- --all at the end just says to apply this filter to every commit.

The git logs now contain the entries that look like following:

commit a0951d56c0d31ef99e0cf9366d0b53cd75bb1073
Author: d62ac7eb9aa769878de91bcd0459b5c5  - <07bd53e7c6c874dfb93d69158135ca9b  >
Date:   Wed Jul 12 16:01:32 2000 +0000

commit b2d8ae2baedc388727e39918af2741e5ba308ebd
Author: 7e521d4c41fa5a518201fb2278acb3c3  - <86a9c9f7d22d7c47f281dc4a68dee371  >
Date:   Tue Jul 11 23:50:22 2000 +0000


A clone of this repository will contain none of the original commit messages or author/committer data, and so will be fine to distribute.

Food in, friendships out

2011-02-21T15:12:00.000-05:00

Wow. I am exhausted, and brimming with hope and cheer and appreciation for all of the amazing people I met this weekend at the Food-in.

The weekend went just as we had imagined and hoped it would be. We didn't have too much in the way of a formal structure set in place for the conference: we hoped simply to rely on the magic that happens when you mash together keen, excited folk with a warm, open space and amazing food. What we got was a veritable frenzy of conversation and idea-sharing and new friendships and excitement.

I was totally blown away by how different each group that we invited was: from the intensely political to the unabashedly epicurean; from those that dumpster-dive for food to those that sourcing directly from farmers; from feeding thirty people once a week to others which feeding four hundred every day; from those that have a formal organisations to those that simply exist by way of friendships and routine; those with two members to those with over a hundred; those that operate soup kitchens to those that operate coffee shops to those that operate bulk food stores; and so on...

And yet, we all had so much to talk about and share. Here is a sample of some of the workshops that were held (we simply created a paper spreadsheet and had people post up workshop/discussions they wanted to run):

Spreading the message: community outreach & publications
Dumpster diving
Food packing and wrapping: serving minimally packaged food
Creating inclusive community with our groups
Maintaining institutional memory
How to organise to win referendums and elections
Community gardening
Food sourcing issues (from farmers and otherwise)
Recipe and cooking techniques exchange
Organising a cross-country solidarity network
Food security outside of the kitchen

Ah-mazing. I'm so grateful for all of the people that came, and to the other Yam!s and peeps that poured a whole lot of love and time into this to make it happen. I'm especially grateful for the folks that drove out from the East coast (twenty four hours of straight-driving!) and Ottawa and Montreal, and to the lone UBC Sprouter who made this part of us reading week adventure. w00t.

Food-in!

2011-02-17T11:35:00.001-05:00

Do you know what has me excited, and kept me wonderfully busy and exhausted these days? A conference this weekend of student-run campus kitchens.

I, and a few other lovely Hot Yam! volunteers, have spent the last few months organising this conference, known as the Food-in: a co-created unconference-style gathering of 60(!) students from across the country converging at the Hart House farm for a cosy weekend to share their story of how their groups started and how they operate, as well as run workshops and discussions for one another, and just generally hang out in the awesomeness.

It's going to be rad.

We've got people from Vancouver, Guelph, Toronto, Peterborough, Lindsay, Kawartha Lakes, Ottawa, Montreal, and Halifax, representing 15 different groups. Look who's coming:

Caffiends. University of Toronto, Toronto
Fleming food initiative. Flemming College, Lindsay
Garden Spot. Carlton University, Ottawa
Peoples Republic of Delicious. University of Ottawa
Glendon Lunik Cafe. York University, Toronto
Happy Belly. McGill's Macdonald Campus, Montreal
Hot Yam!. Do I even need to tell you?
King's Alternative Food Cooperative Association. University of King's College, Halifax
Loaded Ladle. Dalhousie University, Halifax
Kawartha Ecological Growers. Kawartha Lakes (they're our farmers!)
Midnight Kitchen. McGill University, Montreal
OCADSU Campus Cupboard. OCAD, Toronto
People's Potato. Concordia University, Montreal
Seasoned Spoon. Trent University, Peterbourough
UBC Sprouts, University of British Columbia, Vancouver
Teatree House Cafe, University of Guelph, Guelph

You can bet the food is going to be awesome.

Event: Promoting social change through social networks and universities

2010-11-29T14:28:00.000-05:00

From the Psychology Students' Association comes notice of a talk by Dan Dolderman and Romin Tafarodi.

“Like,” Learn, and Launch the Revolution: Promoting social change through social networks and universities

Date: Friday, December 3rd, 2010

Time: 1:00 – 3:00 pm

Location: SS2110

Speakers: Dr. Dan Dolderman and Dr. Romin Tafarodi, Department of Psychology

What do Facebook and U of T have in common, other than sucking lots of hours out of our precious time? The answer lies in their potential to transform our society, namely, to be used as powerful tools for social change.

Social change refers the transformation of cultural and social organizations and structures over time. Our society is never static, and our lives are heavily intertwined with the social, political, economic and cultural changes that occur constantly.

But the question is, how do we promote positive social change? Can we tap into existing tools and institutions, things that are such a fundamental part of our daily life that we, ironically, often take for granted? In this biannual academic seminar organized by the Psychology Students’ Association (PSA), two prominent professors from the Department of Psychology will address these critical questions.

At this seminar, Dr. Dan Dolderman will first discuss how social networking websites can be used to promote positive social change. Next, Dr. Tafarodi will speak about the place of university in envisioning social change. Finally, we will have time for Dr. Dolderman and Dr. Tafarodi to share their insight on each other’s presentations and/or for the audience to ask questions to the two speakers.

Refreshments will be served. Don’t miss out on what is to be a fascinating, mind-opening seminar by two of the best lecturers in psychology on Friday, December 3rd!

Examples of research as activism: some suggestions so far...

2010-11-29T12:38:00.001-05:00

In a recent post I asked for examples of research as activism to inspire crafting a research agenda of my own. Several people have sent me ideas already. Thank you! Here they are:

Ron Deibert, University of Toronto (see also: CitizenLab)
Ronald Baecker, University of Toronto (see also: Technologies for Aging Gracefully)
Michael Geist, University of Ottawa (see also: his personal website)
John Robinson, University of British Columbia (see also: Centre for Interactive Research on Sustainability)
Henrik Moller, University of Otago (see also: Centre for the Study of Agriculture, Food and Environment)
The Western Institute for Social Research has articles on Action research

Jorge Aranda has reblogged the question out to his readers, so check in on his post for other ideas too.

I am going to investigate and blog again about anything interesting I find and other suggestions I get.

McMaster president Patrick Deane's Installation speech

2010-11-26T01:06:00.001-05:00

Last week I attended my sister's convocation from McMaster University. Part of the convocation ceremony involved the "installation" of Patrick Deane as the university's president. He gave a wonderful speech which I reproduce in full below. It is long but worth the read. In it, Deane talks about "the idea of service as an underlying value of the academy" and the importance of the UN's Academic Impact initiative (of which McMaster is a member, but the University of Toronto is not). I particularly like his focus on the importance of universities working with their local community (I've italicised that portion below).

In 1975 I was an undergraduate at the University of the Witwatersrand in Johannesburg. Those were difficult years in South Africa. Persecution and humiliation were the daily lot of the majority of citizens, not least because of a determination on the part of the state actively to contest their right to citizenship itself.

To a young white man from the English-speaking minority, such as myself, the campus environment—at that time a hotbed for anti-apartheid activism—was exciting but also confusing. Confusion arose because the University, being government-funded and owing its existence to parliamentary statute, was in important ways inseparable from (one sometimes thought complicit in) the established order against which many in the student body felt compelled to protest. Predictable intergenerational hostility was complicated by the sympathetic presence at many of our protest meetings of the university Vice-Chancellor, often in full academic dress, and surrounded by eminent colleagues from the professoriate clad in their doctoral scarlet. Although we knew that they wore their academic regalia in an attempt to prevent the police breaking up our meetings as a “riotous assembly,” the spectacle was confusing: here was the leader of the institution, dressed in a manner that asserted his leadership and appeared to confirm his allegiance with the status quo, joining with students in protest against policies propagated by what we thought to be the very source of his own authority, the government.

I have many vivid memories of this time, including one of Dr. Philip Tobias, the eminent professor of anatomy and world renowned authority on human evolution, deftly slipping away through a cloud of tear gas, his billowing robe asserting in very stark contrast to the uniforms of rampaging riot police the elusive yet indomitable value of humane learning and scientific enquiry.

Another memory from the same year is of the annual academic freedom lecture, given by Ronald Dworkin, soon to be famous as a philosopher of law and a constitutional scholar, teaching at Yale, New York University, and the University of London. In 1969, at the age of 38, Dworkin had been appointed to the Chair of Jurisprudence at Oxford, and when I heard him speak in Johannesburg he was a passionate man in his mid-forties. This was a transformational experience for me, although I do not think I realized it at the time.

Dworkin established his reputation as a critic of legal positivism, a doctrine—derived from the work of Jeremy Bentham—that asserts there is no inherent or necessary connection between the validity of law, on the one hand, and ethics or morality on the other. In 1977 he was to publish a book called Taking Rights Seriously, which expanded upon the notion of what he called “law as integrity”—which as an undergraduate listener I took to mean that there is a necessary, epistemic, connection between law and ethics or morality.

This way of thinking blew away the clouds of confusion for me. The notion that an unethical or immoral law must be opposed, and could be opposed without necessarily vitiating one’s allegiance to one’s country, made it possible for the Vice-Chancellor to engage in protest against the enforcement of apartheid in the academy without necessarily abnegating his authority—without being required, figuratively as well as literally, to cast off his gown of office.

The fundamental question was this: in whose name did he wear that gown? At one level this was easy to answer, for as the presence of the ceremonial mace at convocation made clear, the right of the university to grant degrees came from the state. From what source, however, did he derive his authority to critique the state itself, and did all of us—students, faculty and staff alike—derive our right to demur at injustice? If that authority depended upon the government of the day, it was obvious that he and we had no serious right to protest, and our activities were illegitimate. But they did not feel illegitimate, and Dworkin’s insistence on “law as integrity” showed us why: the authority upon which the university was built derived not from the monolithic state as temporarily constituted by partisan politicians, but from society as the constantly shifting sum of human experiences, aspirations and contradictions.

The triumph of reason and justice which was the release of Nelson Mandela and the advent of democracy in South Africa is known to you all, but had we not—had the country not during those dark years—maintained its allegiance to “law as integrity,” the outcome most certainly would have been disastrous. Law had to be understood as enjoying an epistemic, even if in practice inconsistently realized, relation to ethics. Furthermore—and this is simply an extension of that last point—the authority asserted by the Vice-Chancellor in my anecdote was nothing if not derived from values that transcended the state itself. “Law as integrity” was a theory that made not society itself but social idealism—the valorization of a just society—the foundation, justification, and raison d’etre for the law.

That, at least, was what I understood in the 1970s to be the import and application—in South Africa, at that time—of Ronald Dworkin’s thinking. The notion has remained with me ever since, surviving my decisions to leave the study of law, subsequently to become a scholar and teacher of English literature, and eventually to commit myself to the service of higher education in the kind of administrative role into which I have been inducted today. The story of the Vice-Chancellor’s presence at protest meetings is worth telling because it helps make sense of those decisions which, cumulatively, have brought me here to McMaster. The issue of the Vice-Chancellor and his authority, his responsibility to social and human values beyond the immediate and contingent, bears directly upon this ceremony and the significance of the gown that has just been placed on my shoulders.

The gown is a gift of the McMaster University Alumni Association, and I am very proud and grateful to receive it. I am even more proud when I reflect on what the gift means: that our graduates now at work in the world maintain their investment in their university, that they have an interest in its leadership, and that they understand the extent to which the work of the university must be integrated with, or at least responsive to, their hopes and the constructive aspirations of our society at large. I use that word “integrated” to trigger in your minds a recollection of Dworkin and the idea of law as integrity, for insofar as the academic vestments of my Vice-Chancellor in Johannesburg in some way symbolized the dependence of law on ethics and morality, on positive civil aspirations, I invite you to think of this garment as a symbol of “education as integrity,” by which I mean quite simply the obligation we in the university must acknowledge to work constantly towards the betterment of our immediate community and broader society. In this province and in this country those goals are frequently shared with government, but this is not always or necessarily the case, as the South African example attests, and for that reason it is critical to be clear at all times about the source of university authority, and about the different forms of that authority.

I am speaking this morning in commitment to the idea of service as an underlying value of the academy. Today, at United Nations Headquarters in New York City, Secretary-General Ban Ki-moon is being joined by representatives from academic institutions in more than thirty-five countries formally to launch UNAI, or United Nations Academic Impact, an initiative intended to promote the direct engagement of institutions of higher education in programs and projects relevant to the United Nations mandate, and in particular to the realization of that organization’s Millennium Development Goals: the eradication of extreme poverty and hunger, achievement of universal primary education, the promotion of gender equality and the empowerment of women, the reduction of child mortality, improvement of maternal health, progress in combating HIV/AIDS, malaria and other diseases, and the achievement of environmental sustainability.
McMaster University joined UN Academic Impact soon after its establishment, and is one of a relatively small number of Canadian institutions to be signatories, so it is appropriate that today, while sister institutions around the world are gathered in New York City to reaffirm their determination to improve the human condition through higher education, that we should pause to reflect on the way in which McMaster University contributes to that effort.

“Education as integrity”: a somewhat mundane though nevertheless profound interpretation of the phrase is that the practice of learning and teaching is nothing less than the embodiment of honesty and sound ethics. My faculty colleagues on the platform and in the audience would no doubt expatiate at length on the idea of an ethical pedagogy, and it is obvious that a powerful connection exists between the dominion of honesty and ethics in the classroom, and the rule of those same values in society.

Because the university derives its authority from higher human values and a committed civility, it furthermore goes without saying that the day-to-day activities of the institution need to reflect that commitment. Hence, collegiality and the principle of academic self-governance must be respected and reinforced, as must fairness, openness and transparency, as well as the fundamental principle of academic freedom, which not only allows but encourages dissent and disagreement. It remains astonishing to me—and an odd sort of consolation—that the apartheid regime in those difficult days in South Africa could never quite bring itself entirely to eliminate academic freedom. Even though you had to do so under supervision of a stern and disapproving librarian, you could read Marx’s Kapital or the Communist Manifesto; and the Vice-Chancellor could put on his robe of office and tell the student body why detention without trial was indefensible in a civilized society, and why the doctrine of habeas corpus had to be defended. Barbarism, evidently, is only rarely absolute.
There are also less obvious resonances for the phrase “Education as integrity.” The language suggests wholeness: education as a gathering-in or reconciliation of diverse elements, and we are reminded that this is an activity of the highest order, that it should be available to all and should act for the betterment of all, and that education is diminished in value and effect when it falls out of touch with the full gestalt of human concerns.

What is going on in New York at this moment is an attempt to take such an understanding of education beyond sanctimonious generalization, to make the work of the university more meaningful—and the learning process more successful—through a dynamic and interactive engagement with the very human problems which it seeks to address. Over the last decade universities, especially in the English-speaking world, have participated increasingly in a bloodless marketing discourse, focused on “global citizenship” as the goal towards which they and their students should aspire. But for all this time they have failed effectively to re-negotiate the relationship upon which such aspirations might successfully be built, the link between institutions of higher education and the world which purportedly they seek to serve. To require that students acquire an “international experience” at some point in their degree is admirable enough, but also minimal in what it is likely to contribute either to the student’s development or to the nation visited. What kind of education is it that relegates experience of the broader world to an optional add-on available only in the senior years, or—worse—assumes that experience of the world is unworthy of academic credit and must be postponed until after graduation? What kind of education assumes that “the world” begins—or at least demands to be reckoned with—only once you leave our national borders? And what kind of education leads students to believe that the world exists to provide an arena and a resource for their personal improvement?
In New York today the emphasis is on impact. What appears to be yet another conference on academic internationalization is in fact a significant departure from the discourse so far. The sponsorship of the United Nations at this event is misleading: although many nations are represented, nationhood itself is not directly relevant. The intention is to rally the universities of the world to focus on their obligation to address humanity and its most urgent needs. In certain cases, this obligation and the discharge of it will be transnational, while in other cases it will not. McMaster University, as a signatory to Academic Impact, could certainly boast about our contributions to nursing in Pakistan, or to the treatment of diabetes and related disorders in the Indian and Indo-Canadian population, but we could just as easily focus on the work of our researchers in addressing aging or poverty in Hamilton.

I have recently said that at McMaster we will need to define and refine our understanding of our place in the international context. I do believe, though, that we must understand our international commitments as merely a subset of our encompassing human obligation, and it is in attending to the latter that we will find a firm and clear direction to follow. That obligation is not confined to the Millennium Goals of the United Nations, although they do provide a helpful hook on which to hang a more defensible vision for higher education. Universities are comprehensive, multifaceted organizations, and it is just as important to recognize the complexity of the interface between the university and society as it is the simple requirement that we derive at least one part of our authority from society, and from the human dream of health, prosperity, civility, and cultural fulfillment.

The community in which McMaster is situated is no less complex and rich than the university itself. The City of Hamilton and the university have grown up together, and there is no doubt that we have contributed to each other’s success. When I think about McMaster’s “academic impact,” as per the UN initiative, I do think of Pakistan, India and the many places in the world where our teaching and research has brought benefit; but I come back to Hamilton as the place where the university’s engagement with the world has to begin, always. Just as “education as integrity” presupposes collegiality, honesty and fairness within our university community, so it also commits us as an institution to work for the enrichment and development of a healthy, just and prosperous community around us.

This is unproblematic, if not easy, so long as we understand that education, if it is to have and to serve integrity, must be at least a two-way process. Notwithstanding the nomenclature, the best teacher is the individual most open to learning, and any learner fully seized of her subject will inevitably teach. The object of study is that from which we learn, which observation opens a way to imagining the benefits that will accrue equally to our students and to the community when we explore the potential of service or experiential learning in the social sciences, medicine, engineering, humanities, business or science.

To assert that global citizenship cannot be learned in a local context is simply wrong. Poverty on our doorstep is like poverty four thousand miles away: while the cultural and socio-economic determinants may differ, the nature of the human experience is similar, and it is possible to extrapolate from analysis of the local to shed light on problematic areas of the global. This is the gathering-in and interconnecting function of education as integrity.

In closing, and to illustrate that last point, I have one further anecdote involving my alma mater, the University of the Witwatersrand. This comes from a slightly later phase than that with which I opened: the mid- to late-nineties, after the arrival of democracy in South Africa, and at the beginning of what turned out to be a major transformation in the system of higher education. The Vice-Chancellor of the time—by then a different person from he with the penchant for wearing academic regalia while simultaneously breathing tear gas—was faced with a dilemma not unrelated to that confronted by his predecessor. After fifty years in which the majority of the population had been denied a decent school education, should the university protect its distinguished international reputation by insisting on entrance requirements so demanding that effectively the majority would continue to be excluded? Or should the institution’s global standing be sacrificed, perhaps temporarily, in order to address the immediate needs of a hitherto disenfranchised population?

This was a real and for many a painful predicament, particularly since it pitted progressive social views against academic aspiration, sometimes within the same individual, but ultimately the university acted as if Ronald Dworkin’s argument from 1975 was still being heard. Integrity in education meant that international standing bought by betraying local interest was unacceptable; advancement of the university that directly or indirectly hobbled the community in its quest for prosperity, civility and justice was reprehensible. So, for a decade the university concentrated its efforts on undoing, through education, the legacy of apartheid, after which it emerged once again into the international community of learning.

Today, in Ontario, we face pressure to internationalize and yet, as I have said earlier, it is unclear what should be the proper relationship between that global thrust and our local obligations. The universities daily face the challenge of accommodating the many students who wish to attend and are qualified to attend, and in all of this the talk of accountability and obligation looms large. To understand properly the question of the universities’ statutory commitment and authority, and the relation between that and our moral and ethical obligations, will be fundamental to our prospects for success. Although we do not find ourselves in immediate danger from tear gas and other cruder forms of interference and intimidation, the drift of the university sector is nevertheless not entirely within our control. For that reason education as integrity remains worthy of our vigorous advocacy and defense — how urgently you would only understand if you have seen it under threat.

For today’s graduating students, with whom I am very proud to share in this celebration, my hope is that your time at McMaster has given you a great many experiences as inspiring to you and as influential upon you as those I have recalled today from my own student days. If you are anything like me, the real significance of many of those experiences will disclose itself to you only over time. I leave you, though, with the reminder that just as those who would teach must be open to learning, those who have learned—like yourselves—have an obligation to teach or foster enlightenment, in whatever form of work they find themselves, and in whatever circumstances. Without you, the academy will have little impact. I wish you well and commend to you a life of reflection, sound action, and integrity.

(Source)

The Transition movement

2010-11-25T16:06:00.001-05:00

We're currently discussing the Transition movement in the class I'm taking, CHL7001 Building Community Resilience, and I wanted to share it with you.

The Transition movement is a movement wherein communities openly acknowledge global issues like peak oil and climate change and choose to respond by building their resilience and lowering their carbon footprint, often by radically localising their economy. These communities are called Transition Towns, or Transition neighbourhoods if they are part of a larger city, and they are simply just groups of interested people -- government involvement is not necessary, but accepted.

The idea is that a response to the threat of peak oil and climate change is so urgent that communities can and need to act themselves and not wait for national or international government. From the Transition Network website:

What we are convinced of is this:

    •    if we wait for the governments, it'll be too little, too late
    •    if we act as individuals, it'll be too little
    •    but if we act as communities, it might just be enough, just in time.

The sorts of actions this movement encourages are wide and varied, but the overall goals as I mentioned are to build resilience and decrease dependence on fossil fuels. Narrowly, resilience is the property of community that enables it to continue as it, and the environment around it, changes. So, for instance, in light of peak oil concerns, a resilient community is one that can withstand dramatic changes in oil prices (where "withstand" could mean: continue to feed itself, continue to be civil toward one another, etc.) In light of climate change concerns, a resilient community might be one that could withstand changes in rain fall, summer temperatures, or sea level.

Examples of projects that Transition initiatives take on range from simply meeting up every so often to get to know one another, to urban farming and gardening, to skill sharing and stuff sharing (e.g. tools, car, work), to organising local renewable energy, and so on. The push is on the process: to just get started working together, now, and to head in the direction of localising the economy and community. The Transition Network website has a wonderful discussion of "patterns" (yes, exactly like design patterns) that, I think, helps to illustrate how the movement hopes to move in that direction.

(To find out more about the movement, check out the Transition Network website I've linked to above, wikipedia of course, and the book considered to be the bible of the movement: The Transition Handbook. Update: If you're in Toronto, check out the Transition Toronto hub.)

What I most love about this movement is how intensely positive it is. It acknowledges the big issues but it doesn't dwell on them. Instead, it empowers people to start tackling the issues themselves where possible. Because the movement encourages all "sizes" of actions, and doing so in community rather than alone, I think it's much more likely to succeed in engaging people. And, even without the threat of climate change or peak oil, I think the community building the movement supports is very worthy of doing.

There is one big concern I have with the movement: In all of the writings I've seen so far (which, admittedly, isn't a lot) there lacks a critical analysis of social justice issues. Who is part of the movement? Who is left out? What are the implicit norms and values in the Transition community? What sorts of social inequalities will be replicated without considering all of this? And so on. None of these questions are raised, and no guidance is given on how to navigate them.

Good models of research as activism

2010-11-24T18:02:00.000-05:00

What I am need of are good examples of researchers who are applying their work to further causes they support.

If there is one strand that connects any work that I do it is my desire to to be of some benefit to the world, and for that benefit to have a certain immediacy. I am concerned about issues such as peak oil and climate change that are so broad in their impact, so potentially disrupting, and so urgently in need of address that it is hard for me to think otherwise. For me, being an activist is crucial. If I'm going to continue on in the PhD program I will need to find a way to make my research activism.

I see great potential and privilege in the work of a researcher. My hope is that it is a chance to give the necessary and careful attention to an issue in order to facilitate change, and do it in a way that doesn't necessarily need to make money and can take certain risks that an NGO or citizen group could not. I'm groping to be any more specific than that though -- particularly when it comes to how to apply software research. Lots of hammers; can't seem to find worthy nails yet.

In our position paper on "nail finding" in software research, we discussed the action research method as way to both discover and ground a research program in real-world problems whilst also raising the chances of being of immediate benefit.

What I'm missing, I think, are good examples of researchers that are engaged in action research; folks that are researchers and also activists and have managed to blend the two well. I can't quite see yet how to apply the skills of a software researcher but if I had some examples of how others are doing it I think it would help.

So, know of anyone? I'm more interested in process rather than content, so it doesn't matter the discipline. Of course, if you know of people doing work in computer science I'd like to know.

Here are a two related and interesting articles I came across today:

F. Lau, A review on the use of action research in information systems studies, Proceedings of the IFIP TC8 WG 8.2 international conference on Information systems and qualitative research, p.31-68, January 1997, Philadelphia, Pennsylvania, United States. (Most text available on Google books)
Cancian F. Conflicts between activist research and academic success: Participatory research and alternative strategies. The American Sociologist. 1993 March;24(1):92-106. Available from: http://dx.doi.org/10.1007/BF02691947.

Psychology of Activism

2010-11-22T16:48:00.000-05:00

Here is Dan Dolderman, the fellow teaching the PSY435: Environmental Psychology class I'm auditing, speaking about the psychology of (climate change) activism. It's essentially the entire course in an hour. The video sucks but luckily the audio is usually enough.

Urban Orchard Trailer

2010-11-08T18:11:00.001-05:00

This is from two of my friends, Joel Catchlove and Jeremy Nettle, in Australia:

Wonderful! I'm really proud of them. You can found out more about the Urban Orchard projects here.

A global work party this sunday

2010-10-07T16:15:00.001-04:00

Just a quick note in case you hadn't heard: On Sunday this weekend, October 10 (10/10/10), people of earth are organising themselves to take on environmental actions. Check out 350.org to see what work parties have been organised in your hood, or if you have a cool idea for a work party register it there.

I think this is a wonderful wonderful thing. It totally puts into practice the idea of small changes. Show up, do something, with other people, and start to engage.

I haven't decided whether I'll join in a work party that is already set up in Toronto, or if I'll create my own (I know, last minute). One idea I had was to organise a "meet your neighbours" event where the point is to go around your neighbourhood and introduce yourselves. Something I think we rarely do here. Start the process of building a community.

Any other ideas?

Where I'm at

2010-10-04T16:54:00.001-04:00

A few weeks have passed since I started on this new track. It's been busy as. I wanted to take a moment to place myself on a trajectory: to reflect on what I've done, what I'm up to, and where I'm heading.

I am exploring. I've been following my nose on the topic of mental health and climate change. This has meant having lots of conversations, reading books and papers, and attending classes.

I've been reading the APA report on climate change and mental health. This document is very readable introduction to the relevant psychology research. I've also been working my way through the excellent book Why We Disagree About Climate Change to get a broader sense of what we know about how environmental issues are perceived.

As I've written about, I'm sitting in on Dan Dolderman's class on Environmental Psychology, and I've met with him once briefly to explain my interests and get pointers about where to head next.

I've had some really wonderful conversations with two longtime activists: Bill Darnell and Angela Bischoff. They have given me plenty to think about with respect to what makes for sustainable activism, what some of the barriers are that stop activists from practicing these skills, and who else to speak to about this in Toronto. So, more interviews to come...

I've been in touch with Blake Poland, who teaches the course: Building Community Resilience through the Dalla Lana School of Public Health. The course looks fantastic but I'm not sure whether I'll be able to attend (the class is already overfull). Either way, the readings seem very relevant and it's good to know of other people working in the same space here.

I am unsettled. (But, if you know me, it's nothing new). I'm not sure how any of this is going to connect to computer science. I'm currently operating on the principle that a) exploring this area is really exciting to me so it must be leading somewhere useful, and b) by throwing myself in I'll eventually understand enough that some good ideas which bridge the two areas will occur to me at some point. Sometimes, though, I get anxious about yet again being in this liminal state between doing something useful and looking around.

I am excited. See above. Research has never seemed so worthwhile. What I see is the extraordinary potential of helping out by working on an important issue in a careful and considered way. If only I could find an angle...

I am writing. I'm writing up a chunk of my thesis into a publishable paper. And it's tough to be back here again. I'm experiencing serious writers block and discomfort and all the rest. Bird by Bird has been a helpful companion, and so have chats with Alecia and her recommendation to start sharing my work more.

I am teaching. I am co-coordinating and TAing for the Software Carpentry course this term. This is a much more direct application of my computer skills and gets into my interest in teaching and engaging people. In the past it has been very satisfying to be able to teach scientists to use computers more effectively.

I am Yam!ing. Oh, you don't know about Hot Yam! yet? As well as getting your weekly fix of non-corporate wholesome veggies it's one place on campus (and in the city) that draws together passionate, aware and creative people like no other. I've been working with them for almost four years now and can't see that changing any time soon.

UPDATE: I am helping n00bs. Part of my (as yet unblogged about) summer was spent practicing what I preach and so getting involved with various non-technical and "on the ground" groups to find out how they use computers and what their tech needs are. I'm continuing my work on a very part-time basis with the Kawartha Ecological Growers, Canadian Civil Liberties Association, and the Society for Permanent Recovery.

Well those are the main threads of me, for now. Feedback? Ideas? Books? Papers? People?

The strength of small changes

2010-09-24T16:08:00.000-04:00

It is probably unsurprising to hear that our attitudes and beliefs influence our behaviours, but the converse is also true: our behaviours influence our attitudes and beliefs. This positive feedback loop means making small changes in our own behaviour can lead to much bigger changes because of shifts in our self-identity, perceived control over a situation, habits and attitudes.

On Tuesday I sat in on Dan Dolderman's class on Environmental Psychology (sorry, no better link). Wow. It looks to be a wonderful, insightful and important class that applies what we know about human psychology to taking meaningful action on environmental problems. In this post I'll describe what we learned about the theory of behaviour change, and the applied technique of Community Based Social Marketing.

The theory

The theory of planned behaviour is about the interplay between our intentions and our behaviours. The theory posits that there are three things that influence our intentions: our attitudes towards the behaviour, our sense of control and difficulty of the action, as well as how we understand other people's perceptions of the behaviour. So, for instance, you may value the idea of recycling, see it as something you can do easily enough, and since the rest of your street does it already you probably see it as fine to do in other people's eyes, and all of this means that you may choose to recycle.

The key insight here is that all of this works the other way around too: our behaviours influence our attitudes, beliefs, and values.

attitudes, beliefs, values <--> behaviour

This means that changing your behaviour by, for instance, making a habit of recycling, reinforces your perception of yourself as a person that recycles and other behaviours now appear possible that once seemed difficult or unnecessary. Taking on those behaviours further reinforce your self-identity and habits and so leading to even more possible changes in behaviour.

So what?

For environmentalism, it is this positive feedback loop that means small changes in "green" behaviour can lead to a whole shift in what actions a person takes in their daily life, to how they understand and value the environment. Cool!

As well, I think understanding this phenomenon is the antidote to the type of defeatist response to taking any action -- the whole "what does it matter if I change my light bulbs, it's not going to really matter... ". Behaviour change psychology tells us that regardless of the isolated environmental impact of a single action just getting started somehow pays future dividends in making it more likely you'll continue to make environmentally wise choices.

The take away for me here is that we should be vigorously encouraging and supporting people as they make small changes to their lifestyles. Those are the changes that matter even though the big changes loom, and seem more important and equally infeasible.

Raising awareness is only one factor...
The other aspect of this phenomenon we discussed is how many of the popular strategies activists, NGOs, and governments take to change other people's behaviours is based on raising awareness of an issue and providing solid information about the problem and what to do about it. These strategies really only influence our attitudes towards a behaviour (i.e. by trying to make us care about the problem enough to change). The theory of planned behaviour shows us that our attitudes are only one influencing factor and, most importantly, Dolderman says that it is a weak one. There are a whole range of possible ways we could be trying to affect change if we go beyond thinking about changing attitudes and start to think about how to influence our perceptions of control and societal norms.

That is an exciting thought to me.

One example of a more effective strategy -- and I believe the class is going to be centered around this method -- is the, poorly named, technique named Community-Based Social Marketing (it just sounds slimy to me). CBSM is about starting with direct contact among people in a community, and tackling those changes in behaviour that are most likely to succeed (they may not be the biggest ones). The success of a change builds social capital and capacity in the community for further change. I suppose it's like the positive feedback loop that occurs individually as we change our behaviours, only on a community scale. I don't know much more about CBSM, but I'll be sure to report back as the class continues.

Climate change and mental health

2010-08-23T19:21:00.001-04:00

In one version of my story I'd start off describing myself as the child of caregivers. My father is a retired child and family therapist, my mother a nurse, my step-mother a social worker and teacher, and my step-father... well, okay, he's the exception: he was a city property inspector... not much caregiving there. So 3/4 the child of caregivers.

At the conference on climate change last weekend I gave some thought to what aspects of responding to climate change really interests me and where best to put my energy. I think I am slowly realizing that, like my parents, what fulfills me is helping people: being of some benefit in relieving suffering. I am a "cool hand on a fevered brow" sort of person. I don't have much in the way of experience in helping people medically, but at every moment I find myself constantly aware of, and concerned with, how people are feeling and coping emotionally and mentally.

As I've become more and more involved with environmental and social issues, like climate change, I've noticed a few ways in which psychology and mental health needs could be to be addressed, and I'm absolutely fascinated by them. Here are a few of the thoughts I had at the conference with respect to psychology and mental health (note, I don't have much of a background in psychological so I don't have the proper terminology to use yet):

On the one hand there is immediate, or "situation trauma": If you're a victim of torture, war, or natural disaster, you'll likely experience some form of psychological trauma or distress which may benefit from treatment.
Then there is "vicarious trauma" and the related sorts of trauma that come from engaging empathetically with traumatized people, or even stories about those people. From what I understand, the effects can be very similar to more immediate trauma.
Another step removed from traumatic events themselves is the kind of suffering that comes from engaging with the issues: activist burnout, compassion fatigue, or "green fatigue" (i.e. becoming worn out from trying be as environmentally conscious as possible).
Lastly, I see how people deliver and react to news about climate change research, pleas for action, or simply in conversation: from green guilt to the shrug-and-ignore response to outright denial that there is a problem. As I see it, speakers who don't give their listeners any healthy ways of engaging with their issue may being doing more harm (or much less good) than if they had crafted a presentation that empowers people, not only informs them. Similarly, I'll bet there are lots of basic skills we could all learn to help us recognize coping strategies like guilt and denial and choose to react differently.

On that last point, the best presentation I've seen for giving the audience useful ways of responding to the talk was one on peak oil by Michael Lardelli way back in 2005. And the end of the talk, Michael had the usual Things-You-Can-Do slide (i.e. personal actions that lower your own carbon footprint), as well as other slides that pointed out the importance of thinking differently about yourself and the future. Things like: "Start to think about your current car as the last car that you'll own", or "Think of air travel as for emergencies only". These kinds of statements really hit home for me. They aren't hackneyed and ignorable thought-stopping cliches like "reduce, reuse, recycle." They are suggestions for how to change your perspective; they start you on path and give you control and responsibility.

So, recognizing my interest in this area, but also my ignorance, where do I go from here? I'm not entirely sure, yet. But, here are some of the leads I have:

Angela Bischoff: an activist in Toronto who has lectured on mental health and sustainable activism. I contacted her during my work to organise the Power Shift Canada conference last year.
Bill Darnell: one of the founders of Greenpeace who I also contacted for Power Shift. As of last year he was also doing work on sustainable activism.
A few years back I attended a workshop called "From private despair to public action" put on by two trauma experts on PTSD and vicarious trauma in activists. I can't remember their names, nor can I find any info about the workshop online so I'll have to do some digging.

Update: As Nora points out in the comments, the two presenters were Dr. Joan Simalchik and Dr. Yaya de Andrade. Here's the only remaining blurb I can find about the workshop.
The American Psychological Association has set up a task force that has issued a report on psychology and climate change. The Australian Psychological Society has a whole section of their website on climate change and psychology, and Canadian Psychological Association has a section on environmental psychology.

I'd welcome any other suggestions of places to start.

Climate Change Conference, Day 2 & 3

2010-08-15T18:52:00.000-04:00

Phew, I had to leave early on Sunday due to heat exhaustion and brain overload. This conference has been great for meeting people and raising good and tough questions around climate change. Even so the presentations have, at times, been familiar or dry -- and I'm useless in the face of point-form notes presented in monotone and a hot room.

But some highlights:

Dick Peltier gave a talk on melting ice sheets, what is causing it to happen, and what the implications are (for instance, when the west antarctic ice sheet melts sea level will rise in the northern hemisphere because the ice sheet exerts such a large gravitational pull that it holds significant sea water to the south). What I found most fascinating about this talk was Peltier's detailed but clear explanation of how scientists know what the know about the melting through their use of Satellite gravity measurements, and how they correct for land that is still rising since the weight of ancient glaciers melted away. Oh right, and it's also pretty fucking scary to consider the implications of imminent sea level rise, but being scared is somewhat the new normal when you're learning about climate change.

Jim Prall spoke about his work to gauge the credibility of climate change deniers by building a database of climate scientists, their position on climate change, and their citation index. In a nutshell, his results suggest that most of the scientists who are climate change deniers haven't published all that much in the way of climate science compared to the those who accept climate change (that's an understatement, check his very cool visualisation of the results). Prall also gave a brief rundown of important books to read, some thoughts on why people are unconvinced about climate change, and pointed out the excellent resource for debunking climate change myths, skepticalscience.com.

John Streicker spoke about his work in northern Canada to make plans to adapt to the changing climate. "In the north, climate change is here", he said. I was so fascinated by his talk and his pictures of the north that I stopped taking notes, unfortunately. He spoke about the sorts of change that is happening (for instance, ice melting three months earlier than expected -- which is a real issue for the people who rely on it for hunting and fishing), and some of the work he is doing to create a carbon neutral community so that -- and I really love this -- they will have the moral authority to ask others to reduce their emissions. Also, I am now in love with the north.

Ronald Plain spoke about the Aamjiwnaang First Nation community near Sarnia and the terrible pollution they have to deal with, the effects of climate change, and the actions they've taken. But actually, his talk wasn't about that; it was about the need to respectfully engage in activism; it was about how people and organisations with the good intentions to help end up taking poor actions because they don't listen to the people they are trying to protect. But, I'm not doing his message any justice: I was again paying too much to his speech to take many notes.

Adrianne Mugnatto-Hamu spoke to the question of whether nuclear power is a feasible alternative to fossil fuels. Her main point was the insight, taken from Joshua Pierce and Mark Jacobson, that the very high cost and very long time to construction (10-19 years) makes switching impractical. Instead, she suggests that we invest the money (including the money the government pays in liability insurance for nuclear power plants) into renewable energy sources (wind being one of the best options in terms of cost and time to construction).

Lastly, Ralph Torrie gave a really excellent talk (both content and style) on the history of our attempts to change energy policy during the 1970s energy crisis, and how the economic policies used to get out of that crisis do not work for the current energy crisis, but we're using them anyhow! The essence of his point is that in the 70s the solution was to increase efficiency and reduce demand enough to relieve the price pressure, but the current problem isn't about price at all: it's about the need to reduce absolute carbon emissions. What Torrie says is that ultimately this is an issue that isn't economic so much as it is moral. In other moral issues, like child labour and murder, we don't use economic instruments (like carbon tax or cap and trade) to solve the issue, we use regulation and enforcement and that's what we need to be doing here too. Huzzah.

Climate Change Conference, Day 1

2010-08-14T09:35:00.000-04:00

Here's a quick rundown of what the folks I heard speak on Friday talked about:

Corrina Serda gave a modified Inconvenient Truth-type talk that really only spent two slides on the science of climate change (yay!) and spent the rest of the time discussing solutions. They were all pretty familiar: use renewable energy, don't consume as much, inspire others, etc.

Gracen Johnson spoke about the lessons she'd learned from her work to organise the Fill the Hill event. My favourite part of the talk was when Gracen discussed how growing the movement meant we needed to communicate our thoughts and feelings "in the least obnoxious way". Later on in the evening Ralph Benmurgui brought up the same point at tactful but honest communication, and I'm really hoping someone will talk to it directly this weekend. Overall, I was moved by how positive Gracen is, and although I've heard people talk about the need to create a "sense of possibility", I don't think I understood that need fully until reflecting on this talk. This deserves a separate post (I'm trying avoid tl;dr).

Asfan Chowdhury spoke brilliantly about how the West is in denial as to how much climate change is a political and global security problem. The west, Chowdhury says, sees climate change as an environmental and lifestyle problem -- we must fix the environment by changing how we live; in the east, there is mainly ignorance of the issue. The reality is for many people in the east climate change is a matter of life or death. That it is not seen this way in the west is an example of our denial and failure to deal with climate change as a global problem. I must read more from this fellow.

Bill McKibben spoke about the urgency and need to mobilize to affect change immediately. As well, this was the first time I've heard him talk about civil disobedience directly. He didn't say much other than things are urgent enough now that these sort of topics must begin to be discussed. I'm glad he spoke to this, and I'm hoping others will too. Anyhow, his latest project is organising Global Work Party on October 10th, 2010, which I'll also talk more about later. He gave his talk over Skype, which as I've discussed can be flakey, and it was at times. Boo. But luckily the audience was incredibly patient and forgiving.

Andrew Weaver spoke about the communication of science by scientists, and in particular that scientists have a moral and civil obligation (being publicly funded and all) to speak up about climate change.

Elizabeth May, who attended the entire day, spoke in the evening about the political landscape leading up to Copenhagen and what's happened (or hasn't) since.

Climate Change Conference 2010, Aug 13, 15, 16

2010-08-12T23:23:00.000-04:00

I just found out about the Climate Change Conference happening this weekend at Hart House. It looks to have a terrific line up speakers. One of the big names talking Friday evening, and a fellow whom I think is spot on in his thinking and actions, is Bill McKibben whom you might know from his excellent books and talks (and more recently from this wonderful article). If you'd like to learn about climate change from some of the people in the thick of trying to understand and attack it I think this conference is the place to be.

Notes about my arrest at the Toronto G20

2010-07-13T11:44:00.003-04:00

I meant to post this earlier, but here is a description of what happened from my perspective on the evening of June 26, 2010 through until I was released on June 27, 2010. As I mentioned before, I was volunteering at the time for the Canadian Civil Liberties Association as a human rights monitor. Our task was to act as neutral and independent observers of the protests during the G20 -- to simply witness and record what we saw. Again, I want to stress that the following report is only based on what I experienced and can recall now. I am not referring to my notes whilst writing this (and once I was arrested I was unable to continue taking written notes anyhow), but I am going to try to be as thorough as I can, so I don't imagine it'll be much of an exciting read.

Around 9pm or so on Saturday evening, my monitoring partner and I followed the march of about 300 people south on Yonge Street and onto The Esplanade. A line of police officers dressed in riot gear with what appeared to be tear gas guns formed a line about 300m east of Yonge St. on The Esplanade. The marchers stopped in front of the Novotel Hotel, at the police line, and continued chanting the lines they had been chanting for as long as we had beeb following them that evening: “This is what democracy looks like!”, “Peaceful protest!”, “Who’s streets? Our streets!”, and so on. This continued on for about twenty minutes without the police line or the protesters moving, or anything more than chanting and cheering. The main group of marchers stood on the street, with a collection of others standing on the sidewalk to the north.

From the west out of Scott St., a line of police officers in riot gear formed along The Esplanade, blocking any exit to the west, and marched towards the crowd. The officers in both police lines began marching slowly towards each other, clattering their batons against their shields, and so forcing the crowd together. My partner and I were on the north sidewalk by the west line of police. A few people who had been standing outside of the KEG restaurant smoking were also being pushed along by the advancing police line. Some of them called out to say they weren't part of the protest. An officer told us to stay against the wall and they would watch out for us.

The police lines stopped when they were about 30m meters apart from each other, the marchers, and anyone else who happened to be on the street or sidewalk, were now boxed in between the two lines. It was around 10 or 10pm, and dark by this point.

The crowd's chants where more urgent, and more often people called out or chanted with “Peaceful protest!” at the police lines. Several times then, and in the time that followed, small groups of officers would suddenly charge into the crowd, grab someone and pull them back behind their line. The crowd didn't seem to be moving in any direction, just standing and chanting.

A fellow who had apparently made his way from the east side of the crowd came over near us to the west line of police and asked to be let out. I didn't hear the officer's response, but I heard the guy say something like, "But the police on the other side told me to come here!". He wandered back towards the east line. (I didn't think about it then but on reflecting about it now I believe what the police where doing here was probably Kettling.)

One marcher called out to the crowd to say that everyone should sit down, which many did. The crowd was much more quiet then. All of us, both on the sidewalk and in the crowd, seemed to be waiting. Calls and chants began saying, “Let us go!” After a while, during a quiet moment a person in the crowd called out, “Okay, I’ve learned my lesson, I want to go home now!” We waited for another hour or so.

People around me were calling their friends and families on their cell phones, or chatting occasionally with people around them. I had to go to the bathroom at this point, so my my mind was mostly focused on that, but I did call into the CCLA several times to let them know what was happening. My partner approached the line of officers to ask about using a washroom, but they shook their head as he approached and didn't say a word. After an hour or more of waiting, an officer spoke to the crowd and announced that if we raised our hands and waited we would all be peacefully arrested. This was the first time I had heard any communications from the police to the group.

Officers from behind line of police would remove people from the crowd, put them in handcuffs and walk them away. As the crowd got smaller the police lines joined to the south forming a semi-circle enclosing the remaining people in the crowd. The crowd stood still and quietly from what I could see and hear. After a while the officers in the line near us started putting down their shields and taking helmets (it was hot after all!).

I waited, standing, for another one and half to two hours or so before being taken from the crowd and arrested. The officer who took me was calm, friendly, and gentle. I explained that I really had to pee at this point, to which he gave me a sympathetic look and told me he had to go too but that there were no washrooms around and that I'd have to wait until I got to the detention centre which would probably be in 10 minutes. He also explained that I'd probably be out in a a couple of hours.

As the officer was putting my personal belongings (backpack, wallet, phone, etc.. ) in a plastic bag, I explained that I was an observer with the CCLA. I showed him the letter explaining the monitoring program to him as well. He read these over and said he couldn't do anything about it. I asked that I be able to keep the letter with me so that I could present this to officers at the detention to explain my circumstances. The officer said he couldn't do that, but that he'd put it at the top of my property bag so that it would be easy to access. I was given a wrist band with a number on it, and a sticker on my shirt with the same number.

I then had my picture taken and was led to the police van. I was there for probably about 20-25 minutes as a few other guys were also led into the van. We could hear girls in the compartment opposite us laughing or calling out to go to the washroom. We sat in the van for a while and then eventually were driven to the detention centre. The ride seemed to take awhile, and even when we arrived we sat for about 20-25 minutes before we were let out. My need to pee was rather urgent at this point, and so, I could tell was that of the girls beside us who were much much more vocal.

We hopped out one after another and had our handcuffs exchanged for the plastic handcuffs that are really just large zip ties. They're much more uncomfortable, I found, and they have long ends that flop around and poke you and anyone nearby. Anyhow, I was led through the detainment centre by an officer to my cell (cell 3). On the way over the officer brought my attention to a notice that was posted explaining that there was surveillance used throughout the place. Sure enough, there was a camera directly above our cell.

The cell is exactly as the pictures show. A roughly 3 by 5 metre cage, with a cement floor, doorless porta-potty and metal bench. Two of the walls of the cage had metal sheeting on them which I gather was to stop us from seeing the other cages around us. There were about 24 other guys in the cell. Some sitting on the bench or floor, most standing. I stood. At this point I figured it'd only be a short time before I'd be released. Other people commented on the lighting and temperature: yes, we were in an enormous warehouse, lit by fluorescent lights, and kept quite cool. I was in a t-shirt and I was rather chilly for the rest of the stay there (which is odd for me, I'm usually too warm in most places). The floor in our cell was covered in a green dust -- which later one of the officers explained was old paint.

Looking out from the cell, to the left was one of the walls of the warehouse, and to the right were two rows of two school portables. Directly in front of the cell was a long walkway with what looked like walk-through metal detectors at the far end. Beyond those I could make out a much longer walkway between even more cells. To the right of our cell, against the nearest row of portables was a long line of metal shelving units filled with property bags. From what I could hear, and later see through cracks in the metal sheeting covering our cage, there were other cages directly to our right and behind.

No one talked much at first, except for two guys who were joking around with each other and occasionally calling out to the officers that walked by. I think someone asked for the time then, because I remember noting that it was 4:30am and being surprised that so much time had past since being arrested (which was at 1-2am) and from when we were initially detained by the police lines (around 10:30pm).

It's really going to be tough to give a play-by-play for the next 16.5 hours: mostly because it was all very similar -- there was nothing really to reckon the time passing. I spent most of the first while just standing, but my feet were already sore from all the walking I had done that day (I started my shift with the CCLA at 11am on Saturday) so I'd sit on the ground for a while to rest, and then stand up again. I couldn't really stretch out or sleep, except by sitting cross legged and leaning over until my body wouldn't fold anymore and then falling asleep that way. I don't think I slept more than 20 minutes in total.

The conversation in our cell was pretty tame. At the beginning we mostly just gave one other looks of sympathy or frustration. Later on we talk briefly, complaining about the pain of the restraints, or to tell our story about why were at the Novotel, and so on. For a good long while (maybe two hours?) one of the fellows, who was a philosophy and physics major, spoke to a few of the other guys about physics and metaphysics and politics. It was a pretty heady discussion and I was grateful for the diversion. Overall though, were a pretty quiet bunch, especially compared to what we heard from the other cells.

Screaming, and cage rattling, mostly. I heard screams from a cell beside and behind us throughout the morning and afternoon that there were 40 people in their cell. If true, it's hard to imagine since with 24 we were already unable to each have enough space to all sit against the wall. People would scream to be let out, or to see their lawyer. I could hear people kick the cage, which would rattle, especially against the sheet metal used to block the view to other cells. A real racket. Often one cell would start kicking and rattling their cage and then I'd hear others start doing the same until it seemed like everyone was doing it and the roar was unbelievable. Except for our cell, of course. Anyhow, this sort of thing would happen often throughout the morning and the rest of the day.

Around us, through the hallways we could see officers coming and going. Coming out of the portables with people (prisoners?) and bringing them to a line forming in front of the metal detectors at the end of the walkway, or going to the portables with people. There were always officers at the property bags on the metal racks in front of our cell, but I couldn't figure out what they were doing. Often I'd watch an officer poke around through the bags on one shelf, move to the next, and then the next, grab a bag and move it to another shelf, and keep poking around. Sometimes they would leave with a bag and other times without. Sometimes they would simply move bags around from shelf to shelf that I had just seen another officer move. It was maddening to watch. Every few minutes an officer holding a sheet of paper would come up to our cell and ask for someone inside, e.g. "Is John Doe 50245 here? John Doe?" Often the names were hard to pronounce so we ended up just asking the officers to read only the numbers to us (it was quicker that way too). If they didn't find who the were looking for in our cell (and they rarely did) I'd watch and hear them move to the next cell and ask the same question. Several times the same or different officer would come back, sometimes after an hour or so, asking for the same person! I overheard two officers talking: "Do you know where Jane Smith 15412 is?" "No, she could be anywhere. Check the overflow area."

A few folks from our cell were removed by about mid morning.. I think we were down then to 17 or 18 guys. I don't think a single one of us left until later in the evening even though officers kept coming to our cell asking if so-and-so was present. This would be frustrating on its own, but was made worse by the fact that every few hours an officer would come up to our cell and thank us for being so calm and tell us that because of that we would be released soon. Eventually folks in the cell, myself included, started to get upset that we seemed to be being ignored or lied to. I believe it was just incompetence and disorganization for the most part.

Guys in our cage would often call out to officers as they walked by and ask them for water, or for the time, or for an explanation of what was happening and when we'd be leaving. Most would give us the time, few would say anything. Occasionally an officer would come over and talk to us, and seem genuinely sympathetic. One time an officer came over and a few guys in the group started to yell at her about having asked for water but not receiving it. She explained patiently that she didn't know anything about that, and that she didn't have to talk to us but was doing it because she wanted to. She stood with us and answered every question we had, but what was frustrating to me was that most of her answers were: "I don't know" or "I know just as much as you do". She didn't know what the system was for "processing" people, she didn't know how long we'd be, she didn't know when we'd get water or food, etc... She seemed genuinely apologetic and sympathetic. A few other officers did the same, but the sympathetic officers seemed to be the exception. Most either ignored us, or laughed at us as they walked by.

Food and water. Much has been written about it. We got a sandwich and a drink about every six hours as far as I could tell - so I received about four "feedings" (as one officer referred to them as). We were given a styrofoam cup full of water and at one feeding a cup of Tang (which we couldn't exchange for water "because we needed the sugar"). The sandwich was a hamburger bun with processed soy cheese (the officers made a point of explaining that) and margarine wrapped in plastic. None of it was good or satisfying at all, but I didn't ever feel hungry or thirsty for very long either (but I also have a very poor awareness of hunger and thirst). Others in the cell were obviously thirsty and called out for water in between feedings but didn't get any. The cell got pretty trashy as the day went on because there was no where to put the garbage.

I'm not sure what else to relate. I have lots of little stories I could share about the time in the cell: the fellow in my cell who spent five hours calling out to officers to have his restraints loosened before anyone got around to it; how we took to spelling out phrases for officers to read by stuffing our cups in the mesh of the cage ("Let us out now", and "why?"); how in the afternoon we collected all of the trash and pushed it out under the cage door into the walkway in a symbol of protest and frustration; how i learned that if you bend the surplus end of the plastic cuffs 76 times it snaps off; how we asked why we had restraints on whilst being locked in a cell, to which the officer replied smugly, "Because you are arrested"; and so on. Mostly, the day just past really really slowly.

The last time an officer came to our cell to tell we'd be out soon, around 6pm, he asked if we were all arrested because of breach of the peace. We actually hadn't been told why, but we assumed so and said yes. He explained that because of this, instead of having to see a staff sargent like everyone else an officer would come to our cell and walk us out, soon, just as soon as they found our papers. I was let out around 9pm.

An officer came to the cell, and walked me down the walkway past the line of people waiting at the metal detectors, past all of the cells in the long walkway beyond (most of the cells were smaller and had one or two people in them) to a room with another cell. My handcuffs were cut off, and I waited there for about 10 minutes. An officer said to us, "I have good news and I have bad news. The good news is that you are not getting charged. The bad news is that it is pissing rain outside." He then explained that if we were arrested at any further Toronto G20 protest we would be charged with obstruction of the police. As well, he pointed out that all of this was being filmed and that it could be used against us in court. One by one we were released. I was taken out of the cell, my picture was taken again, and then I was handed my property bag and walked outdoors to a turnstile at the end of a driveway and released.

Across the street some intrepid and lovely folks had set up a tarp and were cheering to each person released. I was really grateful for that. I pulled on my rain gear under the tarp, and soon after walked away to catch a bus up to the subway to head home. I wish now I'd stayed a bit and thanked everyone who was there, but I wasn't really thinking so well by then: I believe by the time I had made all the calls to my family that night I'd been pretty much awake for 40 hours.

G20 Healing and Legal Consultation -- Tonight 6-9pm, Christie Pits Park

2010-06-29T15:12:00.000-04:00

I'll bet many of you have witnessed some pretty, uh, intense happenings over the last couple of days involving the police and protesters and other citizens. Or you know someone that did. I myself was acting as a human rights monitor for the Canadian Civil Liberties Association during which I was arrested and held for 22 hours in pretty terrible conditions for no good reason I can, as yet, understand.

Anyway, somehow we all need to come to grips with what has happened and channel our anger and frustration into something constructive. I feel like it's darn easy to succumb to feeling hopelessness in the face of something like this, and then deciding the best thing is just to tune it all out.

So, at the risk of losing credibility by seeming too flaky (I kid. You all know I'm both a flake and whatever the opposite of a flake is, right?), I'd like to pass on an invitation to this event that's happening tonight -- the invitation also contains the contact information for the folks that hope to hold future events like these. I don't know any more about the event than what you can read below, but I sincerely appreciate the sentiment behind the idea of gathering people together for a chance to debrief and hear each other's stories, and sort out what we all should do next. If we'd only have events like these in order to discuss our fears and frustrations about climate change and other environmental terribleness...

***

G20 Healing and Legal Consultation

Date:	Tuesday, June 29, 2010
Time:	6:00pm - 9:00pm
Location:	Christie Pits Park
Street:	http://maps.google.ca/maps?hl=en&client=firefox-a&hs=Vnq&rls=org.mozilla:en-US:official&um=1&ie=UTF-8&q=christie+pit+park&fb=1&gl=ca&hq=christie+pit+park&hnear=Toronto,+ON&cid=0,0,11925874417238815952&ei=HqQpTLbILpCMnQeBtfF7&sa=X&oi=local_result&ct=image&
City/Town:	Toronto, ON
	View Map

Description

This past weekend many people dealt with uncalled for police mistreatment and we want to hold the Canadian government and police apparatus responsible for violating our rights, terrorizing Toronto citizens with no good reason, and treating us like criminals in our own city. We all need to come together and use our numbers to hold them accountable.

Come to Christie Pits Park at 6pm tomorrow (on west side of the park at Bloor St. ) and share your experience with one another and with lawyers that want to help us effectively record our stories and collect our statements. We will also have medics and healers present to help us overcome the emotional and physical abuse we've experienced. Some food will be served on site.

It is really really important that we don't let this go away. If Canadians show passivity towards everything that happened, then we will no doubt face this same brutality again, and again. And let's not forget that this type of police brutality has always existed in our city, but in racist and discriminatory ways. During the G20, we saw this brutality experience on a much broader level.

It was inspiring to see over 1500 people marching in the street today, in a protest against police abuse of power. Many had not taken part in demonstrations over the weekend but were outraged by the police's behavior. The oppression is not working, it is encouraging people to stand up!

Harper has proposed that the G20 continue to be hosted in Toronto. He also congratulated the police force for a fantastic job of keeping the streets safe. Will we let him have the last word?

If you have any questions, please call Dave at 647 968 4619.

If you would like to offer help as a healer, facilitator, lawyer, or anything else please call Dave or simply come out to the park!

Everyone welcomed!

If you can't make it down to Christie Pitts Park tomorrow but would like information on healing spaces that will be organized later please email g20healing@gmail.com - there are other groups also offering services and we will hopefully all work together.

If you would like to share your story and support the call for a public inquiry please visit www.g20inquiry.org

Be safe and see you tomorrow.

How to fail successfully at applying software research to climate change

2010-04-25T02:42:00.001-04:00

The Second International Workshop on Software Research and Climate Change (WSRCC) is being held this year at the International Conference on Software Engineering. Jorge, Val, and I teamed up to write a position paper for the workshop to explain our thoughts on how to apply software research to the problem of climate change. More specifically, we wanted to present the types of thinking and approaches which we felt are mostly likely to be ineffective at putting software research to good use in fighting climate change.

The page where the paper is undergoing an open review is linked above, but here's the paper itself:

The paper is a little unusual, as you might guess from the title of this blog post (which is different from the paper title, as I will explain in a moment). Anyhow, instead of stating our points directly we chose to be a little provocative. In our paper we parody the sorts of approaches we think are ineffectual by presenting exaggerated examples of "useful" research, and coming to wacky conclusions about how researchers should conduct themselves. Our point in writing the paper this way was to be, as I say, provocative, and we hoped it would lead people to disagree with us and then start a discussion as to how we really ought to approach this new (for most us software researchers) domain. That is, (speaking for myself here, but I think Jorge and Val would agree) one big reason for writing this paper was to get people discussing the how rather than the what of research directions -- a sort of call-to-contemplative-action.

The reason I'm saying all of this and not just announcing the paper and leaving it at that, is because I think our approach in writing this paper sorta flopped. It is maybe too subtle, too logical, and isn't very clear about where we switch from earnestness (i.e. climate change is an urgent problem and we can contribute...) to parody (... by treating neural connections as program call graphs and running software metrics on them). The upshot is that several people who have read it failed to see parody, and therefore failed to get our message. Boo...

So how do we fix this? One suggestion we've had is to make it clear from the outset that this is parody by changing the title to something more like the title of this blog post. Another idea would be to turn up the volume on some of our examples, and make even more ridiculous claims... but then, I think, we run the risk of our paper just being seen as a joke paper, which it really isn't.

Hmm... we'd welcome any suggestions (as well as any other comments on the paper, of course!). It's probably best to put your comments over on the workshop blog.

Software Quality in Climate Modelling

2010-04-22T23:07:00.000-04:00

As Greg has already mentioned, this week I completed my Master's research paper: Software Quality in Climate Modelling. Here's the abstract:

A climate model is an executable theory of the climate; the model encapsulates climatological theories in software so that they can be simulated and their implications investigated. Thus, in order to trust a climate model one must trust that the software it is built from is built correctly. Our study explores the nature of software quality in the context of climate modelling. We performed an analysis of the reported and statically discoverable defects in several versions of leading global climate models by collecting defect data from bug tracking systems, version control repository comments, and from static analysis of the source code. We found that the climate models all have very low defect densities compared to well-known, similarly sized open-source projects. As well, we present a classification of static code faults and find that many of them appear to be a result of design decisions to allow for flexible configurations of the model. We discuss the implications of our findings for the assessment of climate model software trustworthiness.

I'd really love to get any comments, criticisms, or feedback of any sort on this work. You can either leave a comment below, or send me email. I have to say, I'm a little anxious about putting this out there what with how charged the discussion around climate modelling has become! That said, I'm excited I finally have something cohesive I can show off and I look forward to your thoughts (especially as we begin to prepare this for publication!).

I'd like to send a big thank you to my advisor Steve, second reader Greg, academic brothers Jorge and Jono, research partner extraordinaire Alecia, and all the other folks that helped to make this happen.

Workshop for highschool students -- Climate change: A call to action

2010-02-19T22:57:00.003-05:00

As my two co-conspirators, Jorge and Val, have already written about, yesterday we ran a workshop for high school students titled, "Climate change and the call to action: How you can make a difference". Here are my thoughts on what we did, what we learnt, and what we could do better next time.

Our objective
The workshop title was given to us, as were some notes and slides from the fellow originally planning to run the workshop. We all but disregarded the notes, and sat down earlier this week to decide what we wanted out of the workshop. Jorge nicely summarises them in his post, but in short: we wanted the students to realise that individual sacrifices are not enough to stop climate change and that they also had to engage with organisations involved in social change to be particularly effective.

Why did we choose this as our objective? Well, for me there are many reasons: "everyone knows" about the personal lifestyle changes we can make -- we've talked about them to death -- but we don't often talk about how to engage with others in our community or, say, with political organisations. As well, I worry about the despair and burn-out that comes trying to take on a problem as big and systemic as climate change through personal lifestyle changes if that is the only way you see yourself having an impact.

A point about the wording of our objective. We initially talked about our objective as explaining that more than just "individual actions" are necessary. After some discussion we decided that no matter whether people are acting to change their own lives or working with others, they are always taking action -- and so the phrase "individual action" isn't very clear. We decided, instead, to use the phrase "individual sacrifice" to refer to choices that affect one's own lifestyle, since many of the choices are often seen as a sacrifice (e.g. give up eating meat.) I won't go into it, but I'm still unhappy with this phrase as I think it unfairly makes out changing your lifestyle to be more sustainable as a negative thing.

The workshop plan
We wanted the workshop itself to be very interactive and, in the spirit of a workshop, get the students to come up with their own ideas on how they could engage with organisations. To do this we planned the session so that we began with an introduction to the topic, and then had the group agree on ground rules for how to conduct themselves through the rest of the activity. Things like: keeping an open mind, taking turns to speak, knowing that whatever was said was to remain confidential, and so on. One point we were careful to include in the ground rules was that this workshop would focus on the actions people could take to fight climate change but that this wouldn't be a place to debate whether climate change was actually happening or not.

We considered having a brief summary presentation to describe what climate change was, how it might affect us, and what influence humans had over the issue, but since we were limited in time (we had only 75 minutes) we opted instead to present them with a simple handout (besides, they were mostly in grade 10, so they probably already knew this stuff):

Then we explained the main workshop activity: a world-cafe-styled series of discussions. The students would break out into groups of 3-5, and discuss a set of questions. After each question, all but one group member would move to a new group to discuss the next question. The remaining group member, the "host", would be there to connect up the discussion from the previous group to the new discussion. We had five questions planned:

What choices or actions can you, or your family, make to help prevent climate change?

GROUP DISCUSSION: What happens if everyone follows the same actions? Would that solve the problem?
Who are the people or groups (the "actors") involved in the issue of climate change in your life?

In your school?
In your neighbourhood, and city?
In your province?
In the country and rest of the world?

What should these actors be doing about this problem?
Are they doing these things? If not, what is stopping them? If so, what is helping them?
What actions will you take as a result of this conversation?

What are the barriers for taking these actions? How do you get around them?

As you see, after the first question, we paused to have a group discussion that was intended to motivate the rest of the workshop: if individual actions -- even if scaled up so that everyone did them -- will not work to stop climate change, then what else will? Question 2 was meant to get the students thinking about who else has power and influence, and questions following that are meant to lead the students to concrete and specific actions they can take to engage with those other actors.

We ended the workshop with an open discussion and debriefing. Yesterday, because we were running low on time, we used Question 5 as the discussion piece. We also gave them a second handout with ideas on how to put their actions into practice effectively (I only have the second page):

My Reflections
As Val has discussed, after the workshop we all had a lot to say about how it went and what we would have done differently. Here are some of my thoughts:

Time: we were far too ambitious for the time we had. To properly do what we have planned we'd need double the time, at least. We could probably have skimmed through the first question, and jumped directly into a discussion about how personal actions alone are not sufficient.

Ground rules: I love the spirit of making ground rules explicit and getting agreement from everyone, but when we did this in the workshop I had the feeling that the students didn't fully understand what they were agreeing to. I think it was because we hadn't explained what the workshop activity was going to be, and so the students didn't know why they were agreeing to the ground rules and what they would need them for. If we were to do the workshop again, I'd consider explaining the activity first, and then have agreement on the ground rules.

Connecting the dots: As Jorge mentioned, we may have reached our goal for some of the students, but I got the feeling that most of them left still thinking about personal sacrifices they could make, and not how they could apply themselves outside of their own lives. Jorge describes one barrier: the system dynamics of the climate make it difficult to really understand the importance of moving beyond lifestyle choices. I think that even without understanding climate dynamics, we still needed to do a better job of getting across the idea that engaging with groups provides leverage for our actions -- that it amplifies our voice and the effectiveness of our actions (as well as being emotionally fulfilling and fun).

What happened in the workshop was that the students listed out actors in Question 2, but mainly focused on those that were large and disconnected from them (e.g. the prime minister, factories, china, multinational corporations, etc...) in the rest of the discussion. I'm simplifying; there was also mention of things like the school environment clubs and councils, teachers, and the school cafeteria. Still, the answers to Question 3 and 4 were mostly big and broad answers. For instance, they listed barriers such as "laziness", "money comes first", "waiting for others to change first", or "personal agenda pushing". Some really great thoughts! But what I think was lacking in the discussion was specificity: specific barriers for actors they had listed, and specific actions they could take with those actors to help overcome those barriers.

If we were running the workshop again I'd consider adding a step where only two or three actors from question 2 where considered in the rest of the questions as a way to keep things focused. If we had the time, I'd also like to see the groups use the strategic planning process we gave them on the handout to experiment with and discuss possible action plans and their merits. Which brings me to...

"But no one listens": One of the students spoke up during the debriefing and complained that writing letters to a politician has no effect, and so it wasn't worth doing. I think responding to this complaint is very important.. it gets to the heart of our objective since we are trying to push people to act out and engage with others -- and why do it if no one listens anyway? I agree! Val suggested that we could present examples of student groups that were making change, as specific examples and also inspiration. If we had lots of time, I think we could make space for a discussion on the various forms of engagement and activism (including, my favourite, the more taboo forms like, civil disobedience -- how fun would that discussion be!).

Slowest. Writer. EVAR.

2010-02-02T00:42:00.003-05:00

That's me. Or, at least, that's how I feel these days. I've spent the bulk of my days this last month mainly writing my thesis, along with finishing up the last of my interviews. Writing has, so far, been very frustrating.

I think I've fallen into just about every trap there is, but primarily the following things trip me up.

I try to write my final draft first. I wouldn't consider myself a perfectionist, but writing certainly can bring it out in me. I'm reminded of this little poem (ahem, "grook") by Piet Hein:

GROOK ON LONG-WINDED AUTHORS
Long-winded writers I abhor,
and glib, prolific chatters;
give me the ones who tear and gaw
their hair and pens to tatters:
who find their writing such a chore
they only write what matters.

I only wish this were me except that I don't think all my perfectionist struggling means I end up writing only what matters... I just get stuck spinning in circles endlessly rewriting sentences. I've been trying instead to simply write my thoughts out in whatever awkward and kludgy fashion they arrive in, but it's hard to do. It's so compelling to pause and rework a sentence or paragraph so that it gets the point across more precisely. Yet doing that, I'm finding, just leads to frustration and then to me losing interest in what I'm doing. When I described this situation to a friend she said something like, "Relax. Of course the first draft sucks, that's why you have 8th and 9th drafts."

I have a restless mind. This is new for me. I'm used to being able to concentrate on a task for long period of time. In fact, I usually have the problem where I forget to eat, drink, or stretch because I get so engrossed in what I'm doing. Not so with writing this month. Instead, it's taken some serious discipline to just keep my bum in my chair and my eyes on the writing page. I've also been spending time in the school library to avoid the distraction of my lab (even though I use the labcoat trick at times). It also doesn't help that I have many other activities in my life that I can and do turn my attention towards. Many thanks to my research partner extraordinaire who has been doing a great job of keeping in me in line.

I'm trying to write it all. I'm pretty invested in this master's thesis at this point. I've put plenty of time and effort into gathering the data for this project, and I now feel a real pressure (from myself, my colleagues, advisor, research subjects, university, tax payers of Canada, you name it) to make sure what I write captures all of that -- that I'm thorough.

I take it personally. When I get stuck in my writing I start to lose confidence in myself. I've never had an easy time with writing -- it's always been something I agonize over -- but then writing has never been central to what I do (i.e. programming or growing vegetables). Of course, in the academic world it is central, and so having trouble writing is a real knock against my ego and confidence. It can really get me down.

I've also discovered I'm just plain slow at thinking at times. Even when I allow myself to write in a no-holds-barred stream-of-consciousness style it still takes me what seems like an unreasonable amount of time to figure out what I want to say. And even when I think I know what I want to say, I find I spend a while trying to articulate in sentences that wouldn't shame an 5 year old. Okay, I'm exaggerating.... I think...

On the upside, when I do get a bit of writing down that I feel good about it's satisfying. Really satisfying. I love it, actually. There's something wonderful about getting thoughts down on a page. It's so enjoyable that, if it wasn't so darn frustrating (and if I was actually any good at it) I'd consider writing as a career.

Curious.

2009 AGU Fall Meeting Poster

2009-12-17T20:57:00.002-05:00

This week Steve has travelled down to San Francisco to attend the AGU Fall meeting and will present posters showing off work done by several of us here in the software engineering lab. One of those posters is one I put together for my study, which I reproduce here:

(PDF version)

Like before, I should note that these are still unpublished and hence preliminary results so please treat them as such. Nevertheless, this should provide more of a flavour of my study and findings to date.

The major changes from the previous poster are the detail in the findings. For this poster I gathered the defect data (from the project repositories) over several revisions, rather than presenting a snapshot from one version of each of the models. I show the maximum, minimum, and average defect density across several versions of each model. This is slightly better I think than showing a single version but it is not completely without problems because the model versions included here differ in status across the models. For instance, in one model I have a repository snapshot that includes several major releases of the model. In another, the repository does not include any major releases but only internal releases. In remaining repository I have only a few major releases and many internal releases. So, what I've presented is a bit of mixed comparison but I believe still gives some insight into the relative defect densities.

The other bit of detail I've added to this chart is a comparator. This data comes from a study by Basili and Green, 1994 with more detailed figures in this presentation by Basili. These figures represent the defect densities by way of reported defects (i.e. bug reports/tickets) , whereas the results I present from the models count defects by way of fixes made. I did not present the reported defect count in this poster but I can tell you that across the board these counts are significantly lower. You can see an example of this in the chart I presented in an early poster linked to above.

As well, I have included a more detailed description of the various faults found through static analysis. Many of these faults are a result of the particular ways the code is divided up during the configuration process. This topic is rather complex and deserves it's own post. I should say though that I do not mean for the various models to be compared in terms of fault rates. At least not until I've given some more thought to how to present the fault occurrences in a sensible way. I say this by way of explaining why I have not included fault densities or absolute fault counts.

Finally, I have included a richer set of findings from the interviews. I could have made a poster completely based on what I have learnt by speaking with the climate scientists that develop the models!

The Story of Cap and Trade

2009-12-01T15:29:00.001-05:00

Update: Steve just informed me that David Roberts has a detailed criticism of the content of this video up at grist.org. The following post isn't about the content of the video but about the presentation techniques. I'm certainly not well-informed enough to make judgements on the content.

From the creator of The Story of Stuff, comes The Story of Cap and Trade.

I won't discuss the content of the short video, but I would like to mention that I think the presentation is marvelous. The animations really do a great job of explaining the concepts. Of course, the narrative itself is very strong and clear, but when the discussion turns to offsets and false offsets I really appreciated the visual element.

Also, the little bubbles that appear along the bottom for more information are a great idea. I wish they actually linked to source material rather than only containing a more detailed explanation. Finally, the presentation ends with specific actions. A wise choice since I think many people will end up wanting to do something after hearing this. (Did I follow through with any of the actions? I did, surprisingly! I signed a few petitions I wasn't aware of).

I would have liked links to more discussion or analysis in the film itself. You can, of course, find plenty of links on the website and especially in the annotated script, you just have to jump through a few links to find it all. As well on the webiste (and script) you can find all of the reports to back up the script. Amazing.

I wanted to share this video because I think it's a great study in effectively communicating a moderately complex idea. It's the combination of a strong narrative and actually enlightening animations (not to mention, cute!) along with links to more information and actions. If you consider this animation according to Steve's criteria for usable climate informatics tools it hits almost all sectors dead on.

The only thing it's missing is an interactive exploration component. I think the video is weakest at the end when it suggests alternatives to cap and trade and this is where some interactivity could come in. The alternatives are somewhat vaguely described and it's not at all clear that they are any better (I mean, cap and trade seems like a good idea on the surface what's the say the devil isn't in the details of any of the alternative suggestions!). I'm not exactly sure what an interactive component would look like for a presentation like this, but possibly a very simplified integrated economic-climate model with knobs to try out the various alternatives and see the economic and climatic effects?

The Munk Debates: Climate Change

2009-11-12T20:28:00.002-05:00

Be it resolved that climate change is mankind's defining crisis, and demands a commensurate response.

On December 1st, the Munk Debates feature a debate on Climate Change: George Monbiot and Elizabeth May take on Bjørn Lomborg and Lord Nigel Lawson. Tickets to attend the event live are sold out, but you can sign up for seats at an overflow showing or sign up to watch the debate as a webcast.

Scientific software quality: what would it take to convince software engineers?

2009-11-09T19:42:00.002-05:00

As I mentioned, last week at the CSER conference I presented a poster summarising my study. I got some great feedback about the coherence and integrity of my study, as well a great question: what would it take to convince the software engineering community of anything about the quality of scientific software?

I had about eight people stop by and actually engage in conversation with me. For most people (mostly other students), this was their first time thinking about scientific software as a type of software worth studying on its own. I had to walk slowly through the first slide to explain how computational scientists use software models to do science. I think this motivated the research questions well, since after the first slide most people understood that climate models are being built and used in an entirely different context from, say, accounting software: there isn't the same certainty about what correct output should look like.

I then explained my approach and findings. Almost everyone had one, or both, of the following reactions:

I'm not convinced.

Yup, just the one. I knew this was their reaction because either people would say so, or they would stare at the poster uneasily until I asked them questions about what they felt about the study. Some felt that the whilst the different parts of the study were reasonable in their own right the study didn't "gel" in a way that either built up (or eroded) their confidence in the quality of climate modeling software. They felt just as unsure about it as before I had explained my study and results.

Here's one example. With one fellow, a professor of software engineering (let's call him Mr. B), I pointed to the defect density chart. This one here:

And I asked whether it gave him any insight or feel for the software quality of the models. He said that yes, this chart led him to believe that the Model B and C were of good quality, and model A was not of great quality. When I asked whether it was the absolute defect rates or the comparison between the different models that led him to believe this, Mr. B said it was the relative rates. But then he also agreed that all of the models had defect densities that were "good". Hmmm... wait, what?

After going through the rest of the study, I asked him what he thought of the software quality of climate models and he said that they were "not very good". He maintained this even when I pointed to the defect density chart earlier on. I asked why he felt that, and he said that because there are so many ways for the climate scientists to interpret and represent their results that they may, even unknowingly, present their models as being more accurate than they really are. He felt that the models could be full of bugs and the scientists may never notice.

To be clear, I'm not convinced either way about the software quality of climate models. I'm trying to leave my opinions for later, once I've collected more interview data at least (and I've expressed before how poor I think measures like defect density are at gauging quality). But, one thing I've come to understand about the climate modelers is that they care very deeply about the correctness of their code -- they most certainly do not take it for granted. I pointed this out, to Mr. B, and I explained some of the ways I've learnt that the climate modelers verify and validate their models.

Mr. B was still unconvinced. And so, as I say, neither were many others.

I understand this. I'm not sure that what I've done convinces me that climate modeling software is "good" or "bad" either. But maybe good/bad and high/low isn't a meaningful way to thinking about quality. Still, I have this urge to distill what I've learnt about the scientific software quality into a quantity. I dearly want some objective measuring stick or benchmark to be able to judge/compare/assess software quality. I think this is what the folks at the poster session wanted too, and didn't get from my poster.

Near the end of the conversation, I asked everyone this final question: what would convince you, as a software engineer, that a climate model is of good software quality or not? I asked this question at the CASCON workshop as well. No one had an answer. In fact, most people just dismissed the question with a laugh. Is it that silly of a question? I think it's a great one (but sure, maybe a touch rhetorical).

I've asked a few climate scientists the same question in earnest: what convinces you that climate model software is of good quality or not? The answers have been quite varied. Knowing the history of the model, or the development team, the state of the documentation, whether they've seen the model code or not, and generally how open the development is, are some of the things that factor into their assessment. Defect densities do not.

What I appreciate about the climate scientists' answers is that they focus more on the internals of the development process than they do on metrics. My interpretation of this is that knowing that the modeling group is following the right processes for building the model, and being able to verify that yourself, is a better indicator of the software quality than any defect metric. (I'm of the mind that this is true in general of software quality assessment... but this is quite a new thought for me and I haven't spent much time thinking through the implications.)

But anyhow, what this leads me to is the idea that maybe a more satisfying way of assessing software quality is some sort of "maturity" assessment, like what is done as part of the CMMI process. "Best practices" for developing and verifying climate models could be established by the community to which each modeling centre could be assessed against. From my interviews so far I can already suggest some of the factors that would go into such standard (check the last two slides).

I'd guess that this sort of assessment scheme would be acceptable to climate scientists, but would it convince the software engineering community? If not, what would need to be included in the assessment? Or, is it just a matter of educating the software engineering community as to the nature of scientific software?

Addendum: As Jorge suggested, maybe I received such suspicion at the poster session simply because of the venue. Poster sessions rank at the bottom of the academic credibility scale, so maybe everyone at the poster session is bound to be suspicious.

Vegetables

2009-11-07T22:28:00.003-05:00

Yep, this post is not about research. It's about vegetables. Specifically, vegetables I just received from two of my friends, Tarrah and Nathan. Just two years ago they bought a farm up near Neustadt, Ontario. They are making a go of starting up an organic, mixed vegetable and livestock farm: Green Being Farm (website under construction).

If you know me, you know that I often affectionately talk about working there last summer. And if you know me, you may have also had the opportunity to eat some of the produce grown at their farm last year (those of you in the lab may remember the potato fairy delivering piles of potatoes to your desk last summer), or you may have been one of the lucky folks to get in on their amazing pastured pork, chicken or turkey.

To eat good, local organic food is a real treat. To eat good, local, organic food that friends of mine grew is amazing. I'm so proud of them and what they are doing, and I'm so proud to be eating their produce. I regularly have doubts about the benefit of my research to the world, but I never doubt the benefit of what people like Tarrah and Nathan are up to.

Modeling the solutions to climate change, part I complete.

2009-11-06T15:53:00.002-05:00

A couple of weeks ago I mentioned a side project a few of us in the software engineering group are undertaking. In short, the purpose is to explore the idea of modeling (graphically, not in code) the various solutions to climate change proposed in a couple of recently published books. The objective is not only to model the solutions, but to model them in such a ways to make it easy to explore the differences and similarities between the different solutions. We have restricted ourselves to just looking at wind power, as a way to make this experiment do-able in a shortish period of time.

Our plan has been to look at each book in turn and completing several different types of model for each book: an ER model, a goal model, and maybe a systems dynamics model. Once we've completed those then we'll explore how to relate them to one another.

We began by looking at a David MacKay's book. Last time I posted the ER diagrams and the start to the goal models. Well, we're done the goal models now, so here they are.

This first goal model covers the first chapter of the book. This has very little to do with wind power, but gives us the the actors and some context from which to build out the wind power model.

And here's the wind power model:

Even if you don't understand i* syntax you should be able to follow these diagrams with some success. Again, I should note that these models represent the view of MacKay himself, as interpreted by our motley group.

Having completed these models marks the end of the first stage of this experiment. The next stage is to do make the same types of models for another book in the series.

I have a few observations to make on the modeling process and this project. This is the first time I've ever been involved in formally modeling anything. The process was, on the whole, frustrating and unsatisfying. I would leave each session feeling like whilst we had put up more boxes and arrows on a canvas we were somehow missing the essence of the text. When we made the entity-relationship diagrams we often seemed to be forcing syntax into the labels on the relationships. The goal models seemed equally inadequate albeit in a different way: there were concepts we all wanted to represent (e.g. facts like "typical wind speed is 6 m/s") but there was no way to say this is in i*. Jen, our modeling Guru who ran most of our sessions, would often tell us that what we were trying to say wasn't easily expressed in a goal model or ER diagram and that this frustration was something she was familiar with from her own work. It felt like putting on a one-armed sweater and fishing around with one hand for the other, non-existent, arm hole. This concern of mine is really just me saying I don't think that the modeling languages we are using are rich enough to really capture what is important. Or also that they don't seem to capture it in "the right way"; that they don't "carve nature at its joints" (plato).

I'm open to this one possibility though: maybe this dissatisfaction points to what is actually a helpful and normal aspect of using several different modeling approaches. Each approach on its own may not be able to represent the full meaning of the text, but taken together they may. And also, by constraining the modeling to specific concerns (entities in the ER diagrams, and goals and actors in the i* models) we are actually providing a more clear picture of the text, and in the future, a more clear picture of the differences between different texts. For instance, it may be easier (at the very least in terms of visual clutter) to show the differences between the basic concepts used in the texts by somehow showing differences in the ER diagrams, without having to, at the same time, navigate actors and goals and process concepts.

This is all just speculation, of course. I'm just trying to say, that my uneasiness may just be because I'm unfamiliar with thinking about something in parts like this.

There's another thing I'm unsatisfied with. If our goal is to show the differences between several types of texts, I fear we may be going about doing this in an unhelpful way. I have nothing to base this on other than just a sinking feeling I get. To me, the real work in this challenge is about how to represent and navigate differences is large and complex structures. In some sense, doing the modeling is easy, or at least known. Trying to explain how one model is different from another in a way that is actually helpful and useful... I have no idea. What we don't have is what we want this model "diff" to look like.

One thought I had: we start by writing up our own summary of the differences between two of the texts. That is, we read two of the books on the list and then collectively write up an essay that compares and contrasts the two authors' solutions in as much detail as we think is relevant. Then we model that document. Doing this will give us the goal posts and probably teach us heaps about what and how to make a visual comparison useful.

Validity and soundness in scientific software

2009-11-04T21:24:00.003-05:00

In today's workshop on Software Engineering for Science we spent quite a bit of time discussing the different levels of correctness of scientific software. I was surprised since I had thought some of this was pretty basic stuff. After a bit of reflection I wonder if it isn't because we don't have common terms for these ideas.

To be clear, I'm referring to verification and validation. These activities are summed up by the questions, "Are we building the right thing?" (validation) and "Are we building the thing right?" (verification). Another way of looking at this is that verification is the act of checking that software meets its specifications, whereas validation is checking that software meets its requirements.

This comes up when you talk about scientific software since in many cases the software is supposed to enact a theory or mathematical model. Validation checks that the mathematical model is accurate where as verification checks that the software implements the mathematical model accurately.

Clearly we have words for "verification" and "validation", though I don't remember these words being used much today, or at all. The fact that they aren't commonly used and that we needed to discuss the distinction between these activities is curious to me.

But more so, whilst we have the words to discuss the activities we don't seem to have adjectives to refer to the software itself. (Do we? Tell me if we do.) I suppose we could use the terms "verified software" and "validated software". "Verified" is overloaded though. I immediately want to ask "by whom?", as if the term refers to software inspected and given a stamp of approval by an outside agency. "Validated software" seems okay though.

Borrowing from formal logic, could we refer to the "soundness" and "validity" of software?

Privilege

2009-11-04T20:37:00.005-05:00

This deserves a much more in-depth discussion which I'm not going to go into here. But I wanted to just take a moment to publicly recognise how privileged I feel, and am, in school. Of course it's not just in being at school that I'm privileged.. it's the country I live in, the socio-economic class I am part of, the people I know, my ethnicity, and so on. And school is a whole other level of privilege.

Today leaving the CASCON conference with two of my colleagues I thought again about how damned lucky I am to be a student here. This is truly a luxurious life. I spent today sitting around a table in a warm room talking with other students and professors about whatever the hell interested us at the moment. We talked while we ate our free lunch. (I repeat, we had a free lunch!) After that we went into another room and talked some more. Again, we talked about whatever interested us. At some point we paused to have tea and stretch. Then we returned to talking until we had had enough. A few of us went home together and spent the entire trip discussing ideas for tomorrow. It was a day of ideas.

And that was a day of work. Ah-mazing. When I'm not at a conference I get to spend an entire day at a sunny desk, spending my day as I please, reading, talking to people, making notes to myself, and generally working on projects as I please.

I feel so so lucky and grateful to be here. It's a fullness of feeling which I'm not sure I can explain all that well. The flip side is that I also feel upset at myself for the times when I take this life for granted. I find it easy to do. Take it for granted, I mean. There are times when, to the exclusion of other feelings, I feel worried about my future, or about a deadline, or how my research project might turn out, etc... But, peanuts! I am a king!

I'm not sure why, but I feel compelled to acknowledge and mention this right now. Maybe just as a reminder for myself. But I'd appreciate hearing any thoughts you have on this topic; so use the comments.

CSER poster session

2009-11-02T14:02:00.007-05:00

This week I attended the poster session at the CSER gathering. This was a great thing to do for a few reasons. Just creating the poster helped me pull together some of my thoughts and results so far. In the same vein, just having to pitch my study and explain what I've been up to helped to clarify my thoughts or bring up new questions. Then, of course, there's the feedback and criticism I get from the attendees, and the new questions they raise (intentionally or otherwise). It's also just fun and validating to have people listen to what I've been up to and engage in a discussion about it... makes me feel like I'm doing something worth talking about.

My poster was in the form of nine "slides". Here they are, with a bit of explanation about each of the slides.

	My study is, as you know, still underway. What I'm presenting here are the method and some preliminary results. I wanted to present at CSER because I wanted to hear what other people would say about some of my findings so far, and whether anyone would have suggestions of where to go next.
	As a motivation for my study, consider how the computational scientist qua climatologist goes about trying to learn about the climate. In order to test their theories of the climate, they would like to run experiments. Since they cannot run experiments on the climate they instead build a computer simulation of the climate (a climate model) according to their theories and then run their experiments on the model. At every step there are approximations and error introduced. Moreover, the experiments that they run cannot all be replicated in the real world, so there is no "oracle" they can use to check their results against. (I've talked about this before.) All of this might lead you to ask ...Why do climate modelers trust their models? Or..
	... for us as software researchers, we might ask: why do they trust their software? That is, irrespective of the validity of their theories, why do they trust their implementation of those theories in software? The second question should actually read "What does software quality mean to climate modelers?" As I see it, you can try to answer the trust question by looking at the code or development practices, deciding if they are satisfactory and, if they are, concluding that the scientists trust their software because they are building it well and it it is, in some objective sense, of high quality. Or you can answer this questions by asking the scientists themselves why they trust their software -- what plays into their judgment of good quality software. In this case the emphasis in the question is slightly different, "Why do climate modelers trust their software?" The second, and to some extent third, research questions are aimed here. Note how I alternate between using "climate scientist" and "climate modeler" to reference the same group of people.
	My approach to answering these questions is to do a defect density analysis (I'm not sure why I called it "repository analysis" on my slides. Ignore that) of several climate models. Defect density is an intuitive and standard software engineering measure of software quality. The standard way to computer defect density is to count the number of reported defects for a release per thousand lines of code in that release. There are lots of problems with this measure, but one is that it is subject to how good the developers are at finding and reporting bugs. A more objective measure of quality may be their static fault density. So I did this type of analysis as well. Finally, I interviewed modelers to gather their stories of finding and fixing bugs as a way to understand their view and decision-making around software quality. There are five different modeling centres participating in various aspects of this study.
	A very general definition of a defect is: anything worth fixing. Deciding what is worth fixing is left up to the people working with the model, so we can be sure we are only counting relevant defects. Many of the modeling centres I've been in contact with use some sort of bug tracking system. That makes counting defects easy enough (the assumption being that if there is a ticket written up about an issue, and the ticket is resolved, then it worth fixing and we'll call it a defect). Another way to identify defects is to look through the check-ins of the version control repository and decide if the check-in was a fix for a defect simply by looking at the comment itself. Sure, it's not perfect, but it might be a more reliable measure across modeling centres.
	Presented here is the defect density for an arbitrary version of the model from each of the modeling centres. For persective, along the x-axis of the chart I've labeled two ranges "good" and "average" according to Norman Fenton's online book on software metrics. I've included a third bar, the middle one, that shows the defect density when you consider only those check-in comments which can be associated with tickets (i.e. there is a reference in the comment to a ticket marked as a defect). The top, "all defects", bar is the count of check in comments that look like defect fixes. I have included in the count all of the comments made 6 months before and after the release date. You can see that bar is divided into two parts. The left represents the pre-release defects, and the right represents the post-release defects. As yet, the main observation I have is that all of the models have a "low" defect density however you count defects (tickets, or check-in comments). It's also apparent that the modeling centres use their ticketing systems to varying degrees, as well as they have different habits about referencing tickets in their check-in comments.
	I ran the FLINT tool over a single configuration of, currently only two, climate models. The major faults I've found are about implicit type conversion and declaration. As well, there are a significant (but small) portion of faults that suggest dead code. Of course, because I'm analysing only a single configuration of the model, I can't be sure that this code is really dead. I've inspected the code where some of these faults occur and I've found instances of both dead code and of code that isn't really dead in other configurations. One example of dead code I found came from a module that had a collection of functions to perform analysis on different array types. The analysis was simliar for each function, with a few changes to the function to handle the particularity of the array. The dead code found in this module was variables that were declared and set but never referenced. My guess from looking at the regularities in the code is that because the functions were so similar, the developers just wrote one function and then copied it several times and tweaked it for each array type. In the process they forgot to remove code that didn't apply.
	Unfortunately, I have as yet only been able to interview a couple of modellers specifically about defects they have found and fixed. I have done a dozen or so interviews with modelers and other computational scientist to talk about how they do their development and software quality in general. So this part of the study is still a little lightweight, and very preliminary. In any case, when I've done the interviews I ask the modelers to go through a couple of bugs that they've found and fixed. I roughly asked them these questions. Everyone I've talked to is quite aware that their models have bugs. This, they accept as a fact of life. Partly this is a comment on the nature of a theory being an approximation, but they also include software bugs here too. Interestingly, they still believe that, depending on the bug, they can extract useful science from the model. One interviewee described how in the past, when computer time was more costly, if scientists found bugs part way through a 6 month model run they might let the run continue, publish the results but include a note about the bug they found and analysis about its effect.
	The other observation I have is connected the last statement on the previous slide, as well as this slide. Once the code has reached a certain level of stability, but before the code is frozen for a release of the model, scientists in the group will being to run in-depth analysis on it. Both bug fixes and feature additions are code changes that have the potential to change the behaviour of the model, and so invalidate the analysis that has already been done on the model. This is why I say that some bugs can be treated as "features" of a sort: just an idiosyncracy of the model. Similarily, a new feature might be rejected as a "bug" if it's introduced too late in the game. In general, the criticality of a defect is in part judged on when it is found (like any other software project I suppose). I've identified several factors that I've heard the modellers talk about when they consider how important a defect is. I've roughly categorised these factors into three groups: concerns that depend on the project timeline (momentum), concerns arising from high-level design and funding goals (design/funding), and the more immediate day-to-day concerns of running the model (operational). Very generally, these concerns have more weight at different stages in the development cycle which I tried to represent on the chart. Describing these concerns in detail probably involves a separate blog post.

Morning discussion for the WSRCC

2009-10-26T13:26:00.004-04:00

This morning Jorge and I attempted to attend the Workshop on Software Research on Climate Change via a skype phone call. But Skype wasn't cooperating. So, we had our a own mini-workshop ourselves. The purpose of the workshop is to respond to the challenge, "how can we apply our research strengths to make significant contributions to the problems of mitigation and adaptation of climate change?" But we interpreted the question as, "What can software researchers do to make significant contributions.... ?" As a result, we considered some alternatives that are probably out of scope for the workshop.

Drop out of research. We recognise climate change is an urgent problem and that many scientific research projects have very indirect, uncertain, and long-term payoffs. For the most part, the problem of climate change is fairly well analysed and many solutions are known, but in need of political organisation in order to carry them out. Perhaps really what is needed is for more people to "roll up their sleeves" and join a movement or organisation that's fighting towards this.
Engage in action research/participatory research. If you decide to stay in research then we propose that you ground your studies by working on problems that you can be sure real stakeholders have. In particular, we suggest that you start with a stakeholder that is directly involved in solving the problem (e.g. activists, scientists, journalists, politicians) and that you work with throughout your study. At the most basic level, they act as a reality-check for your ideas, but we think that the best way to make this relationship work is through action research: joining their organisation to solve their problems, becoming directly involved in the solutions yourself. Finding publishable results is an added bonus which is secondary to the pressing need.
Elicit the requirements of real world stakeholders. As you can see from the last point, we're concerned that as software researchers we lack a good understanding of the problems holding us (society) back from dealing with climate change effectively. So, we suggest a specific research project that surveys all the actors to figure out their needs and the place the software research can contribute. This project would involve interviewing activists, scientists, journalists, politicians, and citizens to build a research roadmap.
Green metrics: dealing with accountability in a carbon market. This idea is more vague, but simply a pointer to an area where we think software research may have some applicability. Assuming there is a compliance requirement for greenhouse gas pollution (e.g. a cap and trade system), then we will need to be able to accurately measure carbon emissions on all levels: from industry to homes.
Software for emergencies. Like the last point, this is one rather vague. The idea is this: in doomsday future scenarios of climate change, the world is not a peaceful place. Potentially more decision-making is done by people in emergency situations. This context shift might change the rules for interface design: where say, in peacetime, a user might be unwilling to double-click on a link, or might be willing to spend time browsing menus, but in a disaster scenario their preferences may change. So, how exactly does a user's preferences change in an emergency, and how might we design software to adjust to them?
Make video-conferencing actually easy. This was our experience all through the day:If we ever want to maintain our personal connections without traveling we need to solve this problem. You'd think that we had already solved it, as we have the basic technology already in place. We have Skype, it is just too flakey for relying on for important gatherings. Or, maybe, hotels and conference centres can't deal with the bandwidth demands. Or, maybe conference organisers don't make remote attendance a priority.

Even getting us through the basic technological obstacles may not be enough for a rich conference participation. Simply having a video and audio feed doesn't compare to face-to-face conversations. Maybe it never will, but certainly we can do better?

Position papers from the 1st Intl. Workshop on Software Research and Climate Change

2009-10-25T16:13:00.002-04:00

Tomorrow the First International Workshop on Software Research and Climate Change is being held as part of the Onward! 2009 conference in Florida. Jorge and I are going to attempt to attend the workshop remotely, so wish us luck. I'll be blogging about the experience tomorrow.

To begin, and as a refresher, I thought I'd post a single sentence summary of each of the position papers submitted for this workshop. Position papers were solicited from participants and were to respond to the challenge stated on the opening page of the workshop. In summary, the challenge is: how do we apply our expertise in software research to save our butts from certain destruction due to climate collapse. Or, as Steve puts it, "how can we apply our research strengths to make significant contributions to the problems of mitigation and adaptation of climate change."

In answer to that challenge, the position papers suggest software research should...

"Data Centres vs. Community Clouds", Gerard Briscoe and Ruzanna Chitchya

... tackle the energy inefficiency of cloud computing by investigating decentralised models where consumer machines also become providers and coordinators of computing resources.

"Optimizing Energy Consumption in Software Intensive systems", Arjan de Roo, Hasan Sozer and Mehmet Aksit

... provide the tools and design patterns for building software systems that meet both their energy-consumption requirements and their functional design requirements.

"Modeling for Intermodal Freight Transportation Policy Analysis", J. Scott Hawker

... improve three aspects of decision-making tools (like, say, an intermodal freight transportation policy analysis model): make them easier to use and interact with (HCI-wise); deal with the complexity of the models and the troubles with integrating various existing implementations; as well as (my favourite), make sure the software is built well since most of the folks doing the building are not trained.

"Computing Education with a Cause", Lisa Jamba

... investigate how to involve computer science students in research "toward improving health outcomes related to climate change" as part of the university curriculum.

"Some Thoughts on Climate Change and Software Engineering Research", Lin Liu, He Zhang, and Sheikh Iqbal Ahamed

... investigate how to navigate and integrate knowledge from many different disciplines and perspectives so as to help people communicate and work together; build decision-support, analysis and educational tools for people, companies, and government; build tools for incorporating environmental non-functional requirements into software construction.

"Refactoring Infrastructure: Reducing emissions and energy one step at a time", Chris Parnin and Carsten Görg.

... use insights from software refactoring to develop refactoring techniques for physical infrastructure (energy grid, water supply, etc.).

"In search for green metrics", Juha Taina and Pietu Pohjalainen

... establish a "framework for estimating or measuring the effects of a software systems' effect on climate change."

"Enabling Climate Scientists to Access Observational Data", David Woollard, Chris Mattmann, Amy Braverman, Rob Raskin, and Dan Crichton

... build systems to help climate scientists locate, transfer, and transform observational data from disparate sources.

"Context-aware Resource Sharing for People-centric Sensing", Jorge Vallejos, Matthias Stevens, Ellie D’Hondt, Nicolas Maisonneuve, Wolfgang De Meuter, Theo D’Hondt, and Luc Steels.

... investigate how to use our everyday hand-held devices as sensors to provide fine-grained environmental data.

"Language and Library Support for Climate Data Applications", Eric Van Wyk, Vipin Kumar, Michael Steinbach, Shyam Boriah, and Alok Choudhary

... build language extensions and libraries to make climate data analysis easier and more computationally efficient.

Modeling the solutions to climate change

2009-10-20T09:54:00.003-04:00

For the past couple of weeks a few of us in the software engineering group have been meeting to take up Steve's modeling challenge: we are attempting to model (visually, not computationally) the proposed solutions from several popular books. The idea is to do so so that it's possible (easy?) to compare the differences and similarities between them. Here is the homepage* for the project, which roughly tracks what we're up to. I'm going to summarise our progress so far.

To start off, we narrowed our focus down to just comparing the books by their take on wind power solutions. We began with David McKay's excellent book, Sustainable Energy -- without the hot air.

In our first few meeting we decided to just "shoot first and ask questions later". That is to say, we just collaborative built up a model of the chapters on wind power as we saw fit in the moment, without following any visual syntax and without worrying too much about what to include or what to ignore. The result looked like this:

At the bottom of that picture is our brainstorming about what other aspects to include (the left hand column), the types of perspectives/analysis that McKay uses and that may be useful to include a future exercise (middle column), and the types of differences we expect to see when comparing models (right column).

The next step would have been to come up with the same sort of model for another book, and then start to figure out how best to make the models comparable so that it is visually easy to see the differences and similarities between the various models.

We didn't do that. Instead, we decided to try making a more principled model. Actually, set of models. We decided to construct an entity-relationship (ER) model, and a goal model (i*) for two books and then see about how to go about making those models comparable.

We began with the entity-relationship model. Again, for McKay's book. McKay's book is fairly well segmented into chapters that have back-of-the-envelope-style analysis and others that have a more broad discussion of the actors and issues. In our first attempt shown above, we mainly only modeled the two chapters on wind-power analysis. But if we just stuck to those chapters for the ER and goal models we'd be left with very impoverished models that miss all of the important contextual bits that frame the wind-power discussion. We relaxed the restriction on our wind-power focus slightly so as to include parts of the book that discuss the context. In the case of McKay's book, chapter one covers this nicely.

After our first few meetings we've completed the ER domain model, as well as made a good start on the goal model.

For the wider context (chapter one), we built the following ER model:

This model is a bit of a monster, but I'm told that most models are like that. Other than the standard UML relationship syntax, we have coloured the nodes to represent whether the concept comes from the book directly (blue), or whether we included it because we felt it was implied or simply helpful for clarity (yellow).

Using the same process we created the following ER model for just the two chapters on wind:

As well, we've begun to go back over the first chapter and build up an i* goal model. Here it is so far:

Stay tuned for further updates on what we're up to. I'd suggest that at the moment these models should simply be taken as our first hack. We haven't done any work whatsoever to make them very readable or comparable, for instance.

* I feel like "homepage" is a rather outdated word now. Is that so?

Geoscientific Model Development

2009-10-18T14:40:00.003-04:00

I had an wonderful chat last week with Stephen Griffies from GFDL. It was a fascinating interview that I'll have to blog about over several posts because we just covered so much territory.

One especially interesting pointer Stephen gave me was to a new journal from European Geosciences Union titled Geoscientific Model Development. This is a journal that accepts articles about the nuts and bolts of building modelling software. It is apparently the only journal like it. Most of the other journals that climate scientists publish in will only accept papers on the "science" derived from the use of such models.

For those of us interested in how climate models are developed, this journal will likely be very relevant. What I find particular cool is the transparent peer-review process and open-discussion. This means for a particular article (say, this one on coupling software for earth-system modelling), you can read the paper and the current referee reviews (with the option to submit your own comments).

One issue with the journal Stephen mentioned is that it is currently not listed in any of the major scientific citation indices. Effectively this means that scientists do not get workplace "cred" for publishing in this journal. Thus, there is little motivation to publish even though, as Stephen put it, having a peer-reviewed publication to "rationalise" code and design decisions is essential to ensuring the scientific integrity of the models.

Talk: Climate Change & Psychological Barriers to Change

2009-09-22T18:09:00.001-04:00

This week is Earthcycle at U of T: an environment week with many many great happenings (see the link for more info). In particular there is what looks to be a great lecture on Thursday discussing the recent report on psychology and climate change from the American Psychological Association.

Here's the full posting:

Thurs. Sept. 24
7:00 p.m. – 9:00 p.m.
Lecture Climate Change & Psychological Barriers to Change, with Dr. Judith Deutsch ( Science for Peace) & Prof. Danny Harvey ( U of T)
International Student Centre, Cumberland Room
33 St. George Street

This is, in part, a summary of a major conference by the Report by the American Psychological Association’s Task Force on the Interface Between Psychology and Global Climate Change titled “Psychology and Global Climate Change: Addressing a Multi-faceted Phenomenon and Set of Challenges.”

The study includes sections on concern for climate change, not feeling at risk, discounting the future, ethical concerns, population issues, consumption drivers, counter-consumerism movements, psychosocial and mental health impacts of climate change, mental health issues associated with natural and technological disasters, lessons from Hurricane Katrina, uncertainty and despair, numbness or apathy, guilt regarding environmental issues, heat and violence, displacement and relocation, social justice implications, media representations, anxiety, psychological benefits associated with responding to climate change, types of coping responses, denial, judgmental discounting, tokenism and the rebound effect, and belief in solutions outside of human control.

A copy of the report is available at http://www.apa.org/science

Dr. Judith Deutsch is a psychiatric social worker and President of Science for Peace.

Prof. Danny Harvey is with the Geography Department at UofT, a member of the IPCC, and an internationally renowned climate change expert.

Organized by Science for Peace
Facebook event page:
http://www.facebook.com/event.php?eid=131392308428&index=1

On static analysis

2009-08-31T19:00:00.000-04:00

Last week I got serious about running a thorough static analysis (using Cleanscape's FortranLint) of one of the climate modelling packages I'm studying. It turns out to be trickier than I thought just to get the source code in a state to be analysed because of the complexity and "homebrewedness" of the configuration systems used.

What do I mean? Well, the models I'm studying are complex beasts. They are composed of many of sub-models, and those sub-models themselves are built from sub-sub-models. For example, a global climate model may be composed of an atmosphere model, an ocean model, and a land model. These sub-models are often functioning models in their own right and can often be run separately. And as I say, the sub-models are also built up from various models. The ocean model may have a sea-ice model, a biogeochemical model, and an ocean dynamics model. There may also be different versions of these sub- or sub-sub models being actively developed.

There are also piles and piles of configuration options for each of these components (the models, the sub-models, the sub-sub-models).

Thus, the climate model code shouldn't really be thought of in the singular sense. It's not source code for a climate model, but for an almost infinite number of different climate models depending on which sub-, or sub-sub-models are included in a particular build, and which configuration options are used.

A word on configuration options. The configuration system for some of the climate models I'm looking at are very complex (as you might expect). They include a generous helping of C preprocessor (CPP) instructions to include or remove chunks of code or other files in order to get just the right bits of functionality. As well, there are many makefiles and home-brewed scripts to assemble and ready the appropriate source files for compilation (e.g. move only the files land ice model version 2 files, not version 1 files, and rename them like so, etc..). Of course, there are also plenty of run-time configuration options slurped in from configuration data files (but since that happens after compilation it's not a concern to me when doing static analysis).

The upshot of all of this is that the source code for a climate model isn't shipped in a state that can be run through static analysis. In order for the static analysis tool to do it's job, it needs to be handed the source code in a ready-to-compile state. After all, the static analysis tool is an ultra-picky compiler that doesn't actually do any compilation but instead just spits out warnings about the structure of the code.

(I'm simplifying slightly: both of the static analysis tools I've looked at (FortranLint and Forcheck) both offer the ability to handle some preprocessing statements. Forcheck implemented it's own limited CPP-style preprocessor, and FortranLint will just call cpp for you on the file. Thus, it is possible to hand the static analysis tool code that isn't exactly in a compilable state, but you still need to configure the static analysis tool to do all the preprocessing... and that essentially duplicates the work that's being done by the homebrewed scripts and Makefiles).

The trouble is that getting a snapshot of the code that's ready for compilation isn't a trivial task. The homebrewed scripts and makefiles do a lot of magic as I described above. Somewhere in that magic -- and often not in one nice, distinct stage -- the code gets compiled. That is, no where in the process is there a folder of preprocessed, ready-to-compile files: configuration and compilation are bound up together.

Ideally I'd like to be able to run the configuration/compilation scripts up to the point in which they produce the ready-to-compile code, then run my static analysis tools over the code, and then continue on with the compilation process so that I can be sure that the code I'm analysing is exactly the code is able to be compiled into a working model. That would be the ultimate validation that I'm analysing the correct code, right? (If I were to use the built in preprocessing facilities of the static analysis tools I can never be sure that I've exactly duplicated the work done in the configuration scripts).

Unfortunately, this separation of configuration and compilation can't be done with out deeply understanding and re-writing the configuration scripts. hmmm... That's one option. It's more messy than I'd like it to be, but I might need to do it to remove any doubts about the validity of my results.

The other option I've come up with is a bit more cavalier, but still might be justifiable. It goes like this: redirect all calls to the compiler in the makefiles to a script that simply copies the target file to another location first before doing the actual compilation. The idea here is to intercept right at the point of compilation in order to take a snapshot of only those files that are compiled and when their in their proper configured and preprocessed state.

In fact, since I don't care about actually compiling the model, the stand-in compiler script could simply output an empty file instead of the actual compiled file. (Outputting an empty file is necessary in order to make other steps of the makefile happy and believe some real work was done.) Of course, replacing the compiler with something that doesn't actually do any compilation also requires that another programs in the makefiles that expect real work to have been done (i.e. the archiving tool, ar) must also be redirected to dummy scripts.

The result would be a folder full of ready-to-compile source files that should, in theory, all be able to be compiled together to make the climate model, and thus ready to be fed to the static analysis tool.

Also, in theory, and with less of a deeper understanding of the climate models, I should be able to compile the files I get from this process into a binary file that I can compare to the binary produced by the unadulterated configuration/compilation process in order to validate this hack.

Where I'm at: I tried putting this process in place last week with one of the models. I successfully got a nice pile of source files to analyse. I'm now just dealing with configuring the static analysis tool to handle external dependencies, but I should know soon whether this idea will work or not.

Abstract of my study for the AGU

2009-08-31T17:29:00.005-04:00

I'm submitting an abstract of my study for the Methodologies of Climate Model Confirmation and Interpretation" session at the American Geophysical Union's Fall Meeting in December. This session (either poster or paper) is aimed at exploring the "methodological issues surrounding the confirmation, evaluation, and interpretation of climate and integrated assessment models".

Here's the current draft of the abstract. I've found it a little tricky to write an abstract for work that I haven't yet completed but I've given it a go. I've gotten some excellent feedback from some of my colleagues (big up to: Steve, Neil, Jono, and Jorge) as to how to frame the problem and my "results" (in quotations because I don't yet have concrete results).

On the software quality of climate models
A climate model is an executable theory of the climate; the model encapsulates climatological theories in software so that they can be simulated and their implications investigated directly. Thus, in order to trust a climate model one must trust that the software it is built from is robust. Our study explores the nature of software quality in the context of climate modelling: How do we characterise and assess the quality of climate modelling software? We use two major research strategies: (1) analysis of defect densities -- an established software engineering technique for studying software quality -- of leading global climate models and (2) semi-structured interviews with researchers from several climate modelling centres. We collected our defect data from bug tracking systems, version control repository comments, and from static analysis of the source code. As a result of our analysis, we characterise common defect types found in climate model software and we identify the software quality factors that are relevant for climate scientists. We also provide a roadmap to achieve proper benchmarks for climate model software quality, and we discuss the implications of our findings for the assessment of climate model software trustworthiness.

Feedback on clarity, wording, grammar, framing of the problem and results, etc... are very much welcome.

Workshops at PowerShift Canada

2009-08-17T22:43:00.002-04:00

PowerShift Canada is a weekend-long youth conference on climate change taking place October 23-26, 2009, in Ottawa. It's modelled after the US PowerShift conferences. Over 1000 highschool and university students and other youth will assemble for hands-on workshops and lectures, and then a full day of lobbying action.

I attended the US PowerShift conference and it was a lot of fun and very inspiring. I'm doing a bit of work with the programming committee for PowerShift Canada. We're looking for speakers and facilitators to run workshops and give talks. Specifically, I'd like to ask you all for ideas on who to invite to speak on the following topics:

A climate science backgrounder
Climate modelling 101
An insider's perspective on the IPCC
Communicating the science of climate change
Developing Canada's GHG inventory

I'm also working on fleshing out workshops on Health and Community, as well as more practical skills workshops (e.g. how to be involved in non-violent civil disobedience, how to facilitate a group meeting, how to cope with activist burn-out, etc.).

If you have any suggestions of potential speakers, if you'd like to speak yourself, or if you'd like to suggest workshop topics, send me email at jon.programming@powershiftcanada.org.

Counting lines of code

2009-08-17T21:35:00.001-04:00

I've been using the CodeCount tool to count lines of Fortran code. Here'r some of the gruesome details of what that entails -- for posterity's sake.

In part of my study I'm measuring defect densities of various climate models. Defect density is the number of defects divided by size of the project measured in lines of code (and most often per 1000 lines of code). Thus, I need to be able to count lines of code. Fortran. Often mixed versions. In this blog post I'll describe one of the limitations I've come across in using the CodeCount tool.

The following table summarises the default behaviour of the CodeCount tool on a snippet of Fortran. The Lines column contains the lines of the Fortran and preprocessor code being analysed. Note, this isn't a working piece of code in any way but that doesn't matter to the CodeCount tool. It's just a collection of lines I used to test the tools behaviour. Anyhow, the Type column specifies how CodeCount categorised the line: comment (comm), blank line (blank), executable (exec), data declaration (decl), or compiler directive (comp). The Physical Line and Logical Line columns specify whether CodeCount counts these lines towards the physical and logical line counts, respectively.

Lines	Type	Physical Line?	Logical Line?
!! this is a comment	comm	no	no
	blank	no	no
#if defined foo	exec	yes	yes
#ifdef key_squares	exec	yes	yes
#include "SetNumberofcells.h"	comp	yes	yes
#else	exec	yes	no
#endif	exec	yes	yes
SUBROUTINE A(Sqr_Grid)	decl	yes	yes
USE Sqr_Type	exec	yes	no
IMPLICIT NONE	decl	yes	yes
IF (assoc(cur_grid)) THEN	exec	yes	yes
Type(grid), Pointer :: Sqr_Grid	decl	yes	yes
WRITE(,) &	exec	yes	no
'Hello'	exec	yes	yes
ENDIF	exec	yes	yes
END SUBROUTINE A	data	yes	no

The physical line count is just a count of non-blank, non-comment lines. The logical line count tries to be a bit smart by counting lines in more abstract terms (I imagine a philosopher-computer scientist in some windowed office somewhere chin-stroking and asking, "What is a line of code?"). Anyhow, CodeCount computes logical line count by ignoring lines with continuation characters (e.g. "&") and certain other statements (e.g. "USE", "CASE", "END IF", "ELSE") and by counting each statement in a multi-statement line as a separate line. The full specification is in the CodeCount source if you're interested.

So the question I could ask is: do I use the logical or physical line count? It's a small question but, oh, I went there. The logical line count is appealing in that it seems likely to be more robust across different coding styles, and maybe gets more at the essence of what the size of a program is (whatever that means; see chin-stroking philosopher above for more information).

Unfortunately the CodeCount tool is too smart (or too stupid) in the way that it counts logical lines. It doesn't gracefully handle pre-processor statements or certain Fortran dialects. This you can see from the table above in the two places I've highlighted in red.

As far as I can make out, as long as a line contains only "ELSE" (other than non-word characters) CodeCount counts this line only as a physical line, not a logical line. So, it counts preprocessor lines as logical lines, except in the case of "#else", which it ignores. Should preprocessor lines be counted as lines of code? I don't know, maybe. Probably, in fact. If so, then we should count all of them as logical lines. Unfortunately, from the bit of digging I've done I can't see how to get CodeCount to consider "#else" as a logical line without messing with the code. No thanks.

But, alas, there's more. CodeCount counts an "ENDIF" as a logical line as you can see, but I don't think it should. See, as mentioned, it's built so that it does not count an "END IF" as a logical line. Now, I'm totally new to Fortran but most references I've come across close an IF block with an END IF, but I've seen one or two references to closing an IF block with an ENDIF. And in fact, some of the code I'm analysing uses exactly that syntax. So, CodeCount will have a slightly inflated logical line count if I use it for these source files.

Again, to fix this problem I'd have to resort to hacking the source if I want CodeCount. And, since I'm so new to Fortran I don't even know the extent to which there are differences in the various dialects so even if I were to decide hacking the source was a good idea, I wouldn't ever be sure I'd fixed it completely. (For instance, I just found out there are also "ENDDO" statements, not "END DO", statements in one of my sources!)

In short: I've been sticking to using physical line counts.

Some results from Forcheck

2009-07-17T17:48:00.004-04:00

In this post I'll describe some of what I've found by using Forcheck to analyse climate modelling code. I am currently evaluating it and Fortranlint as tools for the static analysis portion of my study. tl;dr: forcheck took some time to configure, but it was configurable in every way I've needed it to be (with a few minor exceptions) and the results seem to be exactly like what I was hoping for.

I decided to start with analysing NASA modelE climate model because I could easily understand the build/configuration system and navigation my way around code easily enough. Some of the other models I have the source to seem a bit trickier. The analysis I'm about discuss isn't on the entire model source code, but only for one particular configuration of modules (an ocean-atmosphere coupled configuration though, so it includes many of the source modules).

In any case, I'll give you the goods upfront. I'll show you the big long summary of problems found forcheck found, but first let me explain the format. Here is an example item:

    2x[ 84 I] no path to this statement

The "2x" means that this issue was found two times. 84 is the unique issue identifier that can be used to look up the issue in the forcheck documentation. I means that this issue is an informative type issue, as opposed to a W warning, or E error. The rest of the line contains a short description of the issue. Got it? Good.

Here's the big list:

1564x[344 I] implicit conversion of constant (expression) to higher accuracy
635x[681 I] not used
265x[675 I] named constant not used
144x[323 I] variable unreferenced
144x[699 I] implicit conversion of real or complex to integer
125x[ 94 E] syntax error
108x[109 I] lexical token contains non-significant blank(s)
107x[319 W] not locally allocated, specify SAVE in the module to retain data
96x[557 I] dummy argument not used
78x[345 I] implicit conversion to less accurate data type
65x[316 W] not locally defined, specify SAVE in the module to retain data
65x[665 I] eq.or ineq. comparison of floating point data with constant
38x[313 I] possibly no value assigned to this variable
35x[342 I] eq.or ineq. comparison of floating point data with zero constant
34x[644 I] none of the entities, imported from the module, is used
27x[124 I] statement label unreferenced
27x[315 I] redefined before referenced
27x[341 I] eq. or ineq. comparison of floating point data with integer
22x[125 I] format statement unreferenced
21x[  1 I] (MESSAGE LIMIT REACHED FOR THIS STATEMENT OR ARGUMENT LIST)
21x[514 E] subroutine/function conflict
19x[530 W] possible recursive reference
18x[674 I] procedure, program unit, or entry not referenced
18x[598 E] actual array or character variable shorter than dummy
10x[340 I] equality or inequality comparison of floating point data
8x[325 I] input variable unreferenced
7x[347 I] non-optimal explicit type conversion
7x[565 E] number of arguments inconsistent with specification
6x[582 E] data-type length inconsistent with specification
6x[668 I] possibly undefined: dummy argument not in entry argument list
5x[312 E] no value assigned to this variable
5x[691 I] data-type length inconsistent with specification
4x[556 I] argument unreferenced in statement function
4x[621 I] input/output dummy argument (possibly) not (re)defined
4x[384 I] truncation of character variable (expression)
3x[383 I] truncation of character constant (expression)
3x[343 I] implicit conversion of complex to scalar
3x[454 I] possible recursive I/O attempt
3x[570 E] type inconsistent with specification
2x[568 E] type inconsistent with first occurrence
2x[573 E] data type inconsistent with specification
2x[ 84 I] no path to this statement
2x[651 I] already imported from module
2x[617 I] conditionally referenced argument is not defined
2x[214 E] not saved
2x[236 E] storage allocation conflict due to multiple equivalences
2x[700 E] object undefined
2x[307 E] variable not defined
1x[115 E] multiple definition of statement label, this one ignored
1x[145 I] implicit conversion of scalar to complex
1x[228 W] size of common block inconsistent with first declaration
1x[230 I] list of objects in named COMMON inconsistent with first declaration
1x[250 I] when referencing modules implicit typing is potentially risky
1x[667 E] undefined: dummy argument not in entry argument list
1x[676 I] none of the objects of the common block is used
1x[616 E] input or input/output argument is not defined

number of error messages:           200
number of warnings:                 192
number of informative messages:    3415

I've only taken a peek at at a few of these issues in detail. Some of them are nonsense and can be disregarded right off the bat. For instance, the 125 syntax errors? Well, most of them come from the fact that the source files contain the cpp macros __FILE__ or __LINE__ and I haven't figured out yet how to make forcheck expand them (or I haven't worked out the ModelE Makefile magic to get cpp to do it instead).

Looking at the most frequent issues now. The most frequent is casting constant to a higher accuracy. Here's an example from the file QUESDEF.f:

frac1=+1.

where frac1 is defined as a REAL*8. I wouldn't think that casting unknowingly to a higher accuracy would pose much of a problem... but what do I know. Can anyone think of some examples where it would be?

The next most frequent issue is the "not used" issue. The issue here is that a variable has been declared but then never gets used before it goes out of scope. There are two examples of this in the file OCNFUNTAB.f, a module with lookup table functions. From my cursory look, it seems that many of the functions are similarly structured: both in purpose and in terms of documentation and layout. My guess is that in this specific case this evolved from copying an existing function to use as a template for new one, and forgetting to remove the unused declared variable. A code clone. This isn't always the case, of course. In another file with this issue, SNOW.f, it appears to be the result of commenting out code.

This issue is distinguished from the one that is two below it on the list, "variable unreferenced". The variable unreferenced issue refers specifically to when a variable is declared, and a value is set, but the variable is never accessed. This issue also occurs in SNOW.f, in this curious example:

ccc !!! ground properties should be passed as formal parameters !!!
k_ground =        3.4d0    !/* W K-1 m */    /* --??? */
c_ground =        1.d5     !/* J m-3 K-1 */  /* -- ??? */

k_ground is used later in the function, but c_ground never is.

Next up is the "implicit conversion of a real or complex to integer". Here'r a few examples:

QUESDEF.f:     prather_limits = 0.
OCNFUNTAB.f:   JS=SS
RADIATION.f:   JMO=1+JJDAYS/30.5D0

I, of course, have no idea whether these cases are unintentional, or what the effects of these casts are. I would think that it is generally dangerous to cast unintentionally to a less precise number...

Let's look at one last issue, the cryptic (to me), "lexical token contains non-significant blank(s)". Forcheck describes by saying:

In a fixed format source form blanks are not significant. However, a blank in a name, literal constant, operator, or keyword might indicate a syntax error.

Okay, I still don't get it.. but that's because I'm at all familiar with Fortran (yet). Here's one example from RADIATION.f:

     DATA PRS1/      1.013D 03,9.040D 02,8.050D 02,7.150D 02,6.330D 02,
1      5.590D 02,4.920D 02,4.320D 02,3.780D 02,3.290D 02,2.860D 02,
2      2.470D 02,2.130D 02,1.820D 02,1.560D 02,1.320D 02,1.110D 02,
3      9.370D 01,7.890D 01,6.660D 01,5.650D 01,4.800D 01,4.090D 01,
4      3.500D 01,3.000D 01,2.570D 01,1.220D 01,6.000D 00,3.050D 00,
5      1.590D 00,8.540D-01,5.790D-02,3.000D-04/

Each value, "1.013D 03" for example, is flagged as an instance of this issue. In this particular case these are just ways of writing down Double-precision numbers, but normally (I guess?) you wouldn't see the space, but instead a + or - sign. There might be a forcheck compiler option I can flip that will ignore this particular type of issue. I tried looking for other, possibly more significant instances of this issue. I found many that were "just" because of line continuations. That is, a line was intentionally wrapped and so this appeared as a space at the end of the line and the start of the next. Here are a two examples:

    SEAICE.f:   IF (ROICE.gt.0. and. MSI2.lt.AC2OIM) then
ODIAG_PRT.f:   SCALEO(LN_MFLX) = 1.D- 6 / DTS

I had no idea that fortran has such awful syntax for logical expressions. Yes, you read it correctly, the greater-than operator is .gt. and so on. In any case, in the above example in SEAICE.f, the fact that the and-operator is written as ". and." rather than ".and." is what raises this issue. In the ODIAG_PRT.f file, it is the fact that there is a space in the representation of the double-precision number.

This concludes my look at at the forcheck results for now. I should note a few things about how I configured forcheck to actually analyse the results. The most important thing to know is that forcheck can be configured to analyse the syntax according to the quirks of various commercial compilers. For this example I chose to use the Absoft Pro Fortran 90/95 V9 compiler emulation mode because the documentation for ModelE suggested this compiler works well. I had to slightly customise the compiler configuration to enable cpp preprocessing (off by default). I also had to configure forcheck with two cpp "defines" that specified the compiler and target architecture because there were a few places in the source code that had conditional compilation rules.

As mentioned, I didn't analyse the complete source code of the model. Not every module is used for every run configuration. I simply looked at one of the run configuration files that comes with the model, and configured forcheck to analyse only those modules that were specified there. For example, there is a module each for a variety of different resolution configuration: RES_M53.f, RES_M24T.f, etc., as well as several different versions of, what looks like, physics modules: CLOUDS.f and CLOUDS2.f for instance. The run configuration specifics only one resolution module and only one version of the clouds module and so that's what I analysed.

I point all of this out only to be explicit about the limitations of what I'm reporting here. This is just one slice through the code and, as you would expect, the static analysis report is often misleading. Nevertheless, I think this will make for some interesting starting points for discussion about code quality issues.

Climate wars

2009-07-06T21:58:00.005-04:00

Climate Wars is a three-part feature on CBC's Ideas radio show about climate change, how we are and are not dealing with it, and the potential for social and political breakdown because of that. For those of you interested in climate modelling, there is lots of fun stuff about it in here too.

The first part in the series was replayed this evening and I was reminded about how good this documentary is. 'Good' might not be the best word here. How about totally fucking scary? Yes, that.

If you have not already heard it, I suggest you listen it. I urge you to listen to it. It is that kind of documentary. It is three hours in total so it is quite a commitment, but I think after listening to the first part you'll find the time.

You will not come out happy, and you will likely be uncomfortable whilst listening to it. That's a very very good thing. But not something to try to ignore, push aside, or minimise. If, after listening to it, you'd like someone to chat with about it, come find me.

CBC content:

Listen to Part 1 of Climate Wars

Listen to Part 2 of Climate Wars

Listen to Part 3 of Climate Wars

Click on the player to listen.

Direct links to mp3s: Part 1, Part 2, Part 3.

Static analysis software for Fortran

2009-06-26T12:10:00.003-04:00

I started with the list of Fortran tools on fortran.com:

ftncheck. A free, static analyser for Fortran 77 programs only, though there is an effort underway to update ftnchek for Fortran 90.
Forcheck. Not free. Apparently the oldest and most comprehensive static analysis tool. Can handle up to Fortran 95 and some Fortran 2003. There is a trial offered.
Cleanscape FortranLint. Not free. Can handle up to Fortran 95. Interestingly, there is a testimonial for it from NCAR. A trial version is also offered.
PlusFORT. Not free. The folks that make this, Polyhedron Software, also distribute Forcheck.
Understand 2.0. This isn't in the same class as the others, as it only computes code metrics. It also does some nifty code visualisations. A trial version is available. I fed it the ModelE code months ago, and it seemed to handle it without trouble.

There are also many projects referenced that are now defunct as far as I can tell:

QA-FORTRAN from Programming Research Ltd. This is the software Les Hatton used in his T Experiments paper.
Forwarn, from Quibus.
FOR_STUDY from Cobalt-Blue.

I also came across a 1999 technical report from Council for the Central Laboratory of the Research Councils (now Science and Technology Facilities Council) that summarises the tools available then and lists their static analysis features. Much of this is still useful.

Note: these are all separate analysis tools. I didn't look at the analysis that the various Fortran compilers can do. Polyhedron Software has a very detailed looking comparison chart.

tl;dr: Forcheck, Cleanscape FortranLint.

Currently reading

2009-06-12T16:25:00.003-04:00

I haven't posted paper summaries in a while but here's a selection of what I've been reading:

On software quality:

On scientific software:

On empirical studies:

One-page summary of my research plan

2009-06-11T14:07:00.005-04:00

I'm exploring the software quality of climate modelling software. I'm investigating what software quality means for climate modellers, and how we can go about measuring quality and benchmarking it. For example, some of the broader questions that motivate me are: How can we determine the software quality of a climate model? How can we compare the quality of one model to another? How can we compare software quality to commercial products or to other computational science software applications? What do the climate modellers themselves mean when they attempt to build good quality software?

These are big questions. To start to answer them I am going to do two small things. Firstly, I am going to inspect the climate modelling software itself. I will use fault density (i.e. statically identifiable errors and "misuses" of the programming language) as well as bug density (i.e. reported and fixed defects) to benchmark the software quality of several climate models. This analysis will carefully consider these statistics across the various defect dimensions (for example, pre- and post-release and defect type). See my blog post on counting defects for more details.

I believe the more interesting questions about how climate modellers view software quality cannot be answered through their code or bug reports alone. So, I'd also like to interview some of the climate scientists that have built the software. Specifically, I'd like to ask them for the stories behind a selection of defects that they found and fixed during development or after a release. The intention here is that by asking questions about the circumstances of a defect and about the judgements made to find and solve it I may start to piece together an understanding of their specific notion of software quality. See my blog post that describes this part of the study in more detail, and my blog post that describes some of the questions I can ask about defects.

Thoughts? Comments? Questions?

The shape of the playing field

2009-06-10T14:21:00.004-04:00

Ever since my first pitch of the defect density study I've been trying to work out what the bigger research questions are here. I'm not really content with just collecting the defect density results unless I can see how the results fit into a larger story.

My first instinct was to find out more about defects themselves and to see what other people have done with them in their studies. What is the relationship between defect densities and software quality? How do other people understand software quality and measure it? What can we really learn about software quality from looking at defect densities? And to what use can I put these results once I have them? I'm starting to get a picture of defect densities and their usefulness, and it is not nearly as good of a tool as I had thought, but it is still worth evaluating.

The title of my talk last week was one possible framing of a much bigger question: why do climate modellers trust the code they write? As in Daniel Hook's presentation at SE-CSE '09, trustworthiness seems like an appropriate way to frame the discussion about software quality when it comes to climate models. Why? Because, coarsely, in computational science pursuits like climate modelling there are not always hard and fast rules to distinguish correct and incorrect results. As Hook says, there are no perfect oracles to which results can be checked against (the oracle problem), and that even if oracles existed the approximations and measurement errors inherent in modelling can make it tricky to distinguish any introduced error coming from faulty code (this is the tolerance problem).

So, how then do the climate modellers know if they're on the right track when constructing their models? We know they employ a wide suite of sophisticated tests to tease out flaws in the conceptual model (validation) and errors in their implementation of that model (verification). My understanding is that underlying some of the validation work are judgement calls, gut checks, and tacit heuristics used to distinguish whether a model is doing the right thing. For example, climate modellers might ask of a model output, "is it raining where it ought to be raining?" The answer to this question isn't well-defined, but it can be answered with a lot of background knowledge and familiarity with the climate processes. This is partly the oracle problem at play. The model output is the result of a scientific experiment and not something we could hope to give a complete description of before hand. I'm not saying validation is all guesswork -- not even close -- but just that there are unformalisable elements to model validation that, I don't think, we're used to thinking about when we discuss traditional software testing. We are used to thinking about software as having more explicit and testable requirements[1].

On the verification side, the tolerance problem entails that, even if we ignore the conceptual problems with the model, it is still not a straightforward matter to be certain if the code is correct. Uncertainties in the data, truncation error in approximations, and round-off error in computations can all hide real errors resulting from flaws in the model implementation.

Asking why climate modellers trust the code they write is one way of trying to understand what climate modellers are doing when they attempt to write good quality code. Given that they have such radically different notion of requirements and correctness, how is their notion of software quality different? If you can't always write unit against the bulk of your work, and you can't always explicitly write down rules for correctness, what then do you mean by good code? I think it's important to start with these questions because the answers inform other questions about the usefulness of defect densities and of quality benchmarking. With a firmer idea of what quality actually is for climate modellers, we can then work on how best to measure it or benchmark it.

To summarise, the primary question is:

What does software quality mean for climate modellers?

The software quality folks have come up with an impressive list of attributes of software quality known is the "ilities". Maybe a more specific version of the above question asks about which quality attributes are most important for climate modellers.

I think there are two other companion questions that need to be asked:

How do climate modellers judge a piece of code against these quality attributes?

What practices do climate modellers follow to achieve high quality software (in terms of the identified quality attributes)?

If software quality was a game of football, the first question asks about the shape of the field and the rules of the game, the second asks about where the goal posts are, and the third asks about the playbook. Ahem.

So, how I go about answering these questions?

I could ask the climate modellers directly. This assumes that they know the answers explicitly. I'm not sure I could answer the same questions for myself.

I could also look at defects. I've defined a defect before as "something worth fixing". Can we say this means a defect is part of the software, created or omitted, that indicates a lack of satisfaction of the important quality attributes. If so, then looking carefully a defect and its circumstances, and in particular asking the climate modellers questions about reported defects might provide some of the basis for answering the above three questions. Or at least the basis from which to ask more intelligent questions.

That is, investigating why and when a piece of climate modelling software falls short might be the very place to look for exposed notions of quality, quality goals, and the practises used to manage them.

Would interviewing scientists about defects give the complete story? Certainly not. For at least these reasons:

I'd only be able to consider a sampling of the defects, and only interview a sampling of the modellers the defects related to.
As noted in earlier posts, some defects may go unreported. Put another way, the selection of reported defects depends on the type of testing that is done, and not necessarily on the nature of the defect itself. That is, defects are not found if no one goes looking for them.
Refining that point a bit: the defects that are found may only be associated with the subset of the quality attributes that are the least well managed. That is, software may show fewer defects related to quality attributes for which there is a well-functioning process in place. These attributes would not appear to be as well represented and thus may not seem important when, in fact, they are.

[1] I feel pretty strange talking with such authority. Please jump in if you know better.

A framework for counting problems and defects

2009-06-08T14:13:00.000-04:00

Last week I came across this technical report from SEI:

Software Quality Measurement: A Framework for Counting Problems and Defects

Abstract. This report presents mechanisms for describing and specifying two software measures–software problems and defects–used to understand and predict software product quality and software process efficacy. We propose a framework that integrates and gives structure to the discovery, reporting, and measurement of software problems and defects found by the primary problem and defect finding activities....

I haven't yet read through the report thoroughly, though the bits I have read seem immensely sensible. This report doesn't attempt anything too grand. It simply lays out clear definitions, and provides a set of questions to ask yourself when going about trying to understand and count problems and defects. Cool.

In fact, I think these questions will also be great to use when following up with climate modellers about specific bugs. Here they are:

Identification: What software product or software work product is involved?
Finding Activity: What activity discovered the problem or defect?
Finding Mode: How was the problem or defect found?
Criticality: How critical or severe is the problem or defect?
Problem Status: What work needs to be done to dispose of the problem?
Problem Type: What is the nature of the problem? If a defect, what kind?
Uniqueness: What is the similarity to previous problems or defects?
Urgency: What urgency or priority has been assigned?
Environment: Where was the problem discovered?
Timing: When was the problem reported? When was it discovered? When was it corrected?
Originator: Who reported the problem?
Defects Found In: What software artifacts caused or contain the defect?
Changes Made To: What software artifacts were changed to correct the defect?
Related Changes: What are the prerequisite changes?
Projected Availability: When are changes expected?
Released/Shipped: What configuration level contains the changes?
Applied: When was the change made to the baseline configuration?
Approved By: Who approved the resolution of the problem?
Accepted By: Who accepted the problem resolution?

I might add more why and how questions to this list: why did the bug go unnoticed? why is it important to have fixed this bug, at that time? how was the bug fixed? why is the fix appropriate?

A talk to Hausi Müller's group at UVic

2009-06-03T11:44:00.003-04:00

I gave a short talk to Hausi Müller's group here at UVic about my research plans. I've posted the slides so you can take a look. Note the slightly provocative title.

Why Do Computational Scientists Trust Their Software?

View more Keynote presentations from jpipitone.

Some of the feedback I got from the group:

You could try looking at how defect rate changes over time. Use the trends as comparison points, or build a prediction model and then compare parameters of the model. (see ISSTA)
Can you really investigate code quality separately from the concerns of model quality?
How would you (well, the climate scientists) know a perfectly correct piece of software if they had one?
The criticality of a defect has many dimensions.
Consider code churn in estimation of code quality (high churn with low defects -> high quality process?). Or simply just making the churn data available might help ground the defect density data.

I actually enjoyed giving this presentation. Partly because it was to a small group and I'm much happier (read: less nervous) in discussions rather than presentations (I have been known to get incredibly nervous in front of groups). More than anything though, it was helpful to put together the slides as a way to articulate what I'm doing. It's the first time I've felt that I have something resembling a cohesive story to tell.

Counting defects

2009-05-29T16:38:00.006-04:00

I've talked about the issues with using defect counts for judging and benchmarking software quality. I do, however, still think it's worth doing an investigation into the defect counts of climate modelling software because:

It will force me to get my hands into the code and bug reports. My hope is that even a basic familiarity with these things will help me understand issues of quality for computational scientists. It may also give me a bit of currency in discussions with scientists if I have some understanding of the details.
The results may be useful to the individual climate modelling groups, as a gauge for quality within their group.
Doing the study furthers a dialogue between computer scientists and computational scientists and climate modelling groups.
I might end up with something useful!
- As I say in point #1, I might actually gain some insight about computational science software quality. ;-)
- Aside from comparing defect densities, I might find another ways to use this data for benchmarking. For instance, at the workshop for software quality at ICSE, Elmar Juergens spoke about how in judging quality absolute values suck (that might been exactly what he said), and how trend analysis is much better. He was speaking from the point of view of process improvement. But this raises an interesting idea: if we redefine software quality as "a good software development process" (whatever that means) maybe we could use aspects of quality trends as points of comparisons between projects.

As I mentioned in a previous post on Fenton and Ohlsson's work, I'll need to be very specific about how I measure defects to do it. First off, defects come in two major flavours: faults (statically identifiable errors, including dangerous uses of the language) and failures (observable, runtime errors). Most of the papers I've read[1] count defects by looking at bug reports and version control comments. Since a bug report or fix in a repository can cover both faults and failures, a defect in these cases is maybe best described as a problem worth fixing. I've only come across one paper so far that measures faults, and that's Hatton's paper (linked to above, see his slides for more details). Hatton used static analysis software from Programming Research Ltd., which I could run with.

The way that the folks in [1] count defects is simply a matter of counting bug reports, or counting the number of check-in comments that say "fixed", "bug" (or other keywords that suggest a fix for a bug). Some papers count defects before a release is made (pre-release) and others count defects against a release (post-release). What makes a bug pre- or post-release is a matter of opinion: some papers go by how it's marked in the bug database, others set a threshold of days before and after a release date with which to categorise bugs. Some papers explicitly mention that defects are counted only if they have been fixed (i.e. just reporting a defect isn't enough) whereas other papers aren't clear about this. Finally, some papers only consider defects logged against certain areas of the software as worth counting (for instance, an installation problem may not be counted but a UI problem would be). Phew.

I'm sure there are more dimensions I haven't considered!

[1] A sampling: Koru et al., 2007; Fenton & Ohlsson, 2000; Kaaniche & Kanoun, 1996; Zimmerman et al., 2007

Climate Interactive -- Public simulator and Webinar

2009-05-29T16:08:00.002-04:00

This is essentially a RT (What do you call this for blogs? RB?).

You've heard about Climate Interactive before from me, me, me, and Steve -- they build simple climate simulations to be used in political decision-making. They've recently released an even more accessible version of their simulation software:

C-Learn is the more accessible, online version of the C-ROADS simulation, which was recently seen in US State Department Special Envoy Jonathan Pershing's plenary address in Bonn Germany to the UNFCCC. Now you can explore how changes in fossil fuel emissions from three parts of the world, plus deforestation and afforestation, will affect CO2 concentrations, global temperature, and sea level rise. And you can make your own graphs to show others your simulation experiments.

It also means that anyone who wants to run a mock-UN negotiation policy exercise such as "the Copenhagen Climate Exercise" has a simulator to back them up.

Try the sim, give us feedback, and let us know how you are using it! Send it to climateinteractive [at] sustainer [dot] org, or submit your information on our Climate Interactive site.

You can't play with C-ROADS online, but there is an upcoming public webinar to show off and discuss the software on June 3rd @ 11:00am EST. Details:

This session will include interactive demonstrations of C-ROADS (we’ll ask “what if” questions of the simulator), description of past and potential applications, and an open discussion about how open architecture simulators could help contribute to effective policy design and dialogue towards a stable climate, particularly aiming towards Copenhagen in 2009.

On quality in scientific software

2009-05-28T13:56:00.003-04:00

In yesterday's post on the trouble of benchmarking software quality, I gave a list of reasons why just looking at defect counts isn't likely to give us enough information to benchmark software quality.

What I didn't do was give any reasons that are specific to scientific software. For that I'm going to first refer you to slides from Daniel Hook's presentation at the SECSE '09 Workshop. Daniel[1] described scientific software development as a number of model refinements: from measurements to theory to algorithms to source code to machine code. At each refinement there are different types of acknowledged errors due to simplifying assumptions, truncation and round-off errors, etc. There are also unacknowledged errors that come from concrete or conceptual mistakes. Validation and Verification activities attempt to weed-out these errors, but the testing process is frustrated by two problems unique(?) to scientific software:

The Oracle Problem: "In general, scientific software testers only have access to approximate and/or limited oracles". As Neil points out in a comment, the output of scientific software is often what you're looking for -- the results of an experiment. If you knew exactly what you ought to get, you would not be doing science.
The Tolerance Problem: "If an output exhibits acknowledged error then a tester cannot conclusively determine if that output is free from unacknowledged error: i.e., it can never be said that the output is "correct." This complicates output evaluation and means that many test techniques cannot be (naively) applied."

Both of these problems mean that even defining a defect is tricky. It's not like in other domains where you have the notion of a clear set of requirements (however plausible that idea is!) which can be tested against.

[1] Strange, now that I've met Daniel I find it more comfortable to use his first name rather than referring to him as Hook 09. ;-) Maybe also it's because I'm referring to a presentation.

Why I came back to graduate school

2009-05-28T12:05:00.006-04:00

This will be a rather personal post. Earlier this year I made the choice to return to graduate school. I'm damn lucky to even have this choice to make, of course. That fact added a level of pressure to get the decision right.

I had left the Master's program the previous year after completing eight months, and all my courses. I felt, then, entirely at odds with the lifestyle, ineffectual and useless in any research directions, generally disconnected from the activities that keep me sane and feeling worthwhile. Looking back, I was probably also very burnt out after an intense summer season of farm work, and an even more intense semester of school work. It was an incredibly tough eight months.

So why have I come back? (and why did I not choose to continue my farming path, or doing other work with more obvious direct impact?). Here is what I wrote in my journal late last year:

I'm interested in the subject area. My guts are just put together this way I guess.
My work here may or may not have impact, but to some extent the same can be said of any work.
I believe, or want to believe, in the academic process -- that the impact from it can be great ("blue sky" v.s. applied research).
I will get insight by direct experience as to how research works and the academic culture. I hope to answer the question of whether I fit with research work, and if research work fits with me.
I'll get the chance to inspire other CS students and researchers to be more socially relevant. I'll join a, hopefully, curious community and bring with me a different perspective and set of experiences which I hope will be of some benefit.
My work or research may lead me to other worthwhile work which I'm unaware of now.
It's a steady job and a familiar lifestyle from which I can explore other jobs and lifestyles.

Generally, it came down to dropping my expectations. I wrote out these points to summarise:

A master's thesis that doesn't have direct impact does not preclude doing work that does (either alongside my thesis work, or eventually).
My time here may have unforeseeable positive impact in the future, or it may turn me on to more relevant research or work.
Or I may achieve nothing other than spending a few months learning about how other interesting people live.

Finally, with all of that said, I know myself better than to think it would be enough just to realise these things. I wrote down a list of things that I was worried about, and would need to be conscious of -- a list of things I knew I had to be prepared for:

I am in an environment (and a place: UofT) which holds many of my old habits; habits that I no longer find helpful, or like. Being back means having to work to change these habits. Again.
I'm signing up for an indoor, contemplative life. Not the outdoor, action-oriented life I'm used to now. I need a balance, and I'll have to work really hard to get it because this won't be the norm here.
Balance also in my work: both to my research and to other, exploratory work/jobs, and direct-action work. My volunteer and activism work is immediately satisfying and keeps me tethered to sense of reality.
Self-esteem. I'm signing up for work that will/may challenge my skill, diligence, and (as a result) ego. I will need to work hard (the easy part) and work smart (less easy, hopefully I'll learn this) and be supportive of myself (even less easy, but I know how to do this much better now).

On the trouble of benchmarking software quality

2009-05-27T15:47:00.005-04:00

All during ICSE and over the past few days as I settle in at UVic, I've been talking with folks about my research ideas to do with investigating the quality of scientific software. I've had my head in papers on defect densities over the past few weeks, and so this has been a very helpful opportunity to take a step back, and get some discussion going over what exactly my focus ought to be and how to go about the investigation. I'd like to tell the story of where my thinking has taken me -- partly to let you all in on it, and partly as a move to help me organise my thoughts. This may be a bit messy. ;-) And may span a few posts.

As described in the post linked to above, I started out on this research track with the fairly concrete idea of benchmarking the software quality of climate modelling software by using defect densities as a comparison tool. Using defect counts and densities are common way of measuring software quality -- the fewer the defects, the higher the quality. The thinking is this: if climate models turn out to generally be of higher quality than other similarly-sized software projects (measured via defect density) then that's a strong indication that something interesting is going on in the climate modellers software development process -- and we'd better take a further look. If the modelling software is generally worse, that's also interesting and worth investigating (I mean, hey, we software engineers might have something to contribute!).

The glitch is that I don't think defect densities, or many of our quality metrics, are going to be good tools for benchmarking -- they're far too subjective. It's not just me either: everyone I pitched the idea to at the workshop on software quality at ICSE was hesitant about the idea, as were most other folks I talked with at ICSE. Fenton and Neil, 1999 has some discussion about the difficulty of using defect densities for measuring and benchmarking software quality. What it comes down to is:

The way defect data are reported varies across publications. Some papers use defect rate, others defect density, others use failure rate.
When are we counting defects, during pre-release, or post-release, both? Some publications don't mention this.
What constitutes a defect isn't always clear: are we counting statically determinable faults (in which case, what heuristics are we using), or are we considering only those failures found and reported by the developers and users?
A plain count of defects ignores severity. So, how are we accounting for the severity of defects? And who determines the severity? Users and developers might have very different ideas of what the severity of a defect is.
A failure isn't in the code, it's a discovered unsatisfactory behaviour. So how a piece of software is tested and used directly determines the defects that are found (if it's never tested, it appears failure-free!). How are testing effort and usage information (user base size, etc..) accounted for in defect counts?
Finally, different people, teams, and domains have very different ideas of what constitutes good quality (e.g. a usability bug may not be nearly as important for a scientist as it is for a commercial product). Comparing software across these boundaries with a simple defect measure ignores these relative notions of quality, which may mean the measure is useless. (At the very least, the measure must be understood as an indicator of how well a group's software development process works for them rather than an indicator about the software in any objective sense. But maybe this has always been the case.)

It seems that the problem is that beyond just being inconsistently reported and ill-defined, defect density is only a single number standing in place of a very complicated constellation of properties. It may be a useful (and somewhat valid) measurement to use within a project since you could argue many of these properties stay the same from phase to phase (e.g. a team will decide once, at the beginning of project, whether they are counting pre- or post-release defects). But as a benchmark or basis for comparison of quality between projects, I think it's dubious, especially without a better understanding of what we actually want to measure first.

So, how do we get a better understanding? In the interests of experimenting with a one item, one post system, I'll leave that thought for the next post.

Paper: Software quality: the elusive target

2009-05-08T12:26:00.001-04:00

Kitchenham B, Pfleeger SL. Software quality: the elusive target [special issues section]. Software, IEEE. 1996;13(1):12-21.

I'd been looking for an article like this for a while: a good overview of the definitions of quality. Kitchenham and Pfleeger take David Garvin's work on the definition of quality and apply them to software quality. There are five different perspectives to view software quality from:

Transcendental view: software quality can be recognised and worked towards, but never precisely defined or perfectly achieved. Thus, view holds that quality is inherently unmeasurable.
User view: how well the software suits the needs of its users. Measuring quality from a user view involves refining concepts like "reliability" and "usability" into measurable characteristics (say, number of hours of learning time needed).
Manufacturing view: how well the software conforms to its specifications, and development process. Measurement is by defect counts and rework costs.
Product view: how well the software scores on various "internal quality indicators", like program complexity. If the internal quality is high then so must be the external quality.
Value-based view: how much the customer is willing to pay for the software. This view, in some sense, unites the various other views with a very practical measure.

The article talks very briefly about comparing projects by using defect density and scaling by user base, but doesn't go into any of the philosophical troubles. There's also a decent reading list on software quality in general.

Paper: Quantitative Analysis of Faults and Failures in a Complex Software System

2009-05-07T12:28:00.006-04:00

Fenton NE, Ohlsson N. Quantitative Analysis of Faults and Failures in a Complex Software System. IEEE Trans Softw Eng. 2000 August;26(8):797-814.

Fenton and Ohlsson attempt to hack out a bit of solid empirical knowledge amongst the wilderness of published defect data. They note that whilst there are plenty of hypotheses and rules of thumb about defects and defect distribution (e.g. 20% of defects account for 80% of failures (the Pareto principle for software), or large modules are proportionally more reliable than small modules) there is little in the way of published empirical knowledge that can be used for validation or benchmarking. They study defect data from a telecommunication system in order to evaluate the extent to which their data supports or rejects a range of hypotheses about defects.

Fenton and Ohlsson find evidence to support the Pareto principle in software, e.g. the majority of operational defects come from a small number of faults.

They also find evidence that modules with higher defect counts found before release have fewer operational defects after release (and similarly, the inverse is also supported). This point is worth taking slowly. It seems reasonable, on the one hand, that modules with lots of defects before release are simply just undergoing better testing, and so should have fewer defects after release. But, you might also reject this idea and believe that that their are a few poorly designed, troublesome modules that are, and will continue to be, responsible for most of the defects. That is, we can predict buggy modules by looking at where bugs have been found before (this idea is supported, as previously mentioned).

So, exactly how to predict post-release defect density based on pre-release density is in dispute. Okay, sure, so what? Fenton and Ohlsson point out that if their hypothesis is generalisable (that is, if post-release density is lower for modules with high pre-release density, and vice versa) then using defect density as the "de facto" measure of software quality is a highly suspect practice. As they say:

If fault density is measured in terms of pre-release faults (as is very common), then, at the module level, this measure tells us worse than nothing about the quality of the module; a high value is more likely to be an indicator of extensive testing than of poor quality.

So, in terms of a benchmarking tool for comparing projects, pre-release defect density isn't going to work. But how about post-release defect density? The authors show that their post-release defect densities are inline with other published studies, so it's plausable. (I sure hope so, because this has been my plan for benchmarking the software quality of various climate models!)

There's one (at least) missing piece to all of this that I have yet to figure out: what exactly constitutes a pre- or post-release defect? That is, when do you stop counting defects as post-release for release n and pre-release for release n+1? In this paper, it's not entirely clear, but defects were associated with releases (and phases within the release) based on temporal proximity to certain milestones. Again, we've seen some work that associates defects to versions based on their bug report classifications and temporal proximity to releases. It seems to me to be crucial to be able state explicitly what the conditions are for pre- and post-release defects in order for these numbers to be even remotely comparable.

(Ah counting. Back in kindergarten it all seemed so easy.)

Paper: Software faults: a quantifiable definition, J. Munson et al.

2009-05-06T16:30:00.003-04:00

J. C. Munson, A. P. Nikora, and J. S. Sherif, "Software faults: a quantifiable definition," Adv. Eng. Softw., vol. 37, no. 5, pp. 327-333, 2006.

Here's a curious little paper. The authors set out find a rigorous definition of the inherently messy notion of a software fault. Their solution is bold, simple, and ludicrous: just count the number of source tokens that change at each check-in and use that as the number of software faults.

What this paper does have going for it is an extensive reading list on software quality, broken down by subtopic. I'll definitely be coming back to this paper for other reading suggestions.

Paper: Orthogonal Defect Classification

2009-05-05T09:06:00.004-04:00

R. Chillarege, I. S. Bhandari, J. K. Chaar, M. J. Halliday, D. S. Moebus, B. K. Ray, and M. Y. Wong, "Orthogonal defect classification-a concept for in-process measurements," Software Engineering, IEEE Transactions on, vol. 18, no. 11, pp. 943-956, 1992.

Chillarege et al. describe an technique for identifying problem spots in the software development process whilst a project is underway through classifying software defects that come up. They suggest a paradigm and describe a pilot study to validate it, but overall I wasn't convinced. There's a lot in this paper that smacks of advertising over content -- maybe the guts of results are found in all the subsequent papers they link to?

Anyhow, my interest in this paper is for the authors' concept of software quality, their use of defect classification, and their thinking on the link between the two. Plus, it seems to be widely referenced.

To begin, the authors point out a gap in the qualitative-quantitative spectrum of measurement methods we have for software quality. They want a measurement scheme that is lightweight, sensitive to the development process (in that it can help locate process problems), and also consistent across phases of a project and between projects (so those who use it can learn from theirs and others experiences). Statistical defect models (quantitative) and root cause analysis (qualitative) are both done retrospectively, are time consuming, and in the case of the statistical methods, often intentionally ignore the details of the software development process used, so they can't provide detailed process feedback.

Enter ODC (Orthogonal Defect Classification). The main idea: come up with various classifications of defects and map those classes onto the software development process so that every defect points to a process problem. (The word "orthogonal" is used both to mean "mutually independent" and because the authors run the metaphor of software-development-process-as-a-mathematical-vector-space, and defect classes "span" this space).

There are two tricks to doing this. The first trick is to use a layer of indirection in the mapping of defects to parts of process. Defects are first mapped to defect types, and then defect types are mapped to parts of the software development process. Why? Because mapping directly isn't something practitioners can do in the moment (it's error prone, and the attempt to do so is nothing more than a "good opinion survey ... not ... a measurement") and because the indirection allows us to compare results across projects and phases.

My view on this: I'm not sure assigning defect types or mapping a defect types onto a parts of the process is any less error prone or requires any less opinion, it just seems to divide up the opinion-making into smaller chunks.

Anyhow, the second trick is about making sure your defect classes actually span the process space. The authors point out that a sufficient classification scheme is a work in progress that ultimately needs to be empirically validated. A good chunk of the paper is devoted to describing a pilot study of ODC to validate it, or referencing future work.

Looking at defect types in more depth then. The first important point: defect types are chosen by the semantics of the fix, rather than only by qualities of the defects themselves. They are assigned by engineer making the fix. Here are the 8 types of defects:

Function -- errors that effect capability and require a formal design change.
Assignment -- logical errors in small bits of code.
Interface -- errors in interacting with other components.
Checking -- errors in data validation.
Timing/serialisation -- errors in the use of shared/real-time resources.
Build/package/merge -- errors in libraries or change management systems.
Documentation -- errors in documentation and publications.
Algorithm -- efficiency or correctness problems that require fixes through reimplementation (not requiring a formal design change).

The authors investigate how the quantities of defects of each type vary over different phases of a project. They note, for instance, that function bugs appear with greater number in the design phase, and timing bugs appear more in the system test phase. The authors then take this "trend analysis a stage deeper" and provide a correlation table that maps principle defect types to stages in the software development process.

This section is maddeningly vague on details -- it's not clear where the process stages have come from or how the correlations were done specifically. This is a shame because this mapping is crucial to the underlying argument for the usefulness of ODC. Any deviation from the "expected" principle variation trends of defect quantities is considered by ODC to point to process problems, but what exactly the "expected" variations ought to be isn't well described, nor is what exactly constitutes a variation other than to say the judgement is "determined with experience".

Overall I'm suspicious of ODC as described in this paper. Partly because the paper lacks detail, but also because I wonder if the classification scheme is objective enough to work as the authors claim -- especially across projects. (El Emam, K.; Wieczorek, I. 1998) show some evidence that defect classification is repeatable within members of a development team, but even then theirs is a highly qualified experiment.

For my interests, this is the first paper I looked at that hints that it might be possible to compare the quality of two projects by looking at the types of defects that turn up. But it only hints. I'll be posting about other work that adds a lot more murk to these waters.

Paper: Reexamining the fault density-component size connection

2009-04-13T17:05:00.005-04:00

L. Hatton, "Reexamining the fault density component size connection," Software, IEEE, vol. 14, no. 2, pp. 89-97, 1997.

By summarising several studies, Hatton concludes that the number of faults has a logarithmic relationship to code size (or, more generally, code complexity). One implication of this is that smaller components have a higher density of faults (since the logarithmic curve rises sharply at first, and then grows more slowly). The interesting bit of the paper comes when Hatton models these observations from a psychological basis beginning with G. Miller's observation that a person can only cope with 7±2 independant pieces of information in short-term memory. Hatton's model specifies one behaviour (logarithmic) for components as they increase in size up to the "Miller threshold" (Hatton uses a capital Omega prime symbol to denote this), and second behaviour (quadratic) for components as they increase past this threshold.

This paper is simultaneously cool and confusing. Cool because Hatton is trying to rigorously use our knowledge from psychology to explain the underlying causes for defect density, rather than just fit an equation to the data. And he tries to back up his models with lots of data, or clearly state when he can't back it up without doing more empirical work.

But the paper is also confusing for a number of reasons. Some are technical: many of the figures are hard to read: the axis titles are vague or ambiguous --one plot is actually missing an x-axis!; another plot's description conflicts with the description given in the paper text. Others are less technical: Hatton switches from talking about different models throughout the paper and isn't very clear as to the situations he's discussing. I had to re-read the paper a few times because of this (as always, YMMV, maybe it's just me being slow). He's also very vague at points, especially when it comes to explaining why fault density ought to be proportionately higher for components below the Miller threshold,. He says,

"... if a system is decomposed into pieces much smaller than the short-term memory cache, the cache is used inefficiently because the interface of such a component with its neighbours is not 'rehearsed' explicitly into the cache in the same way, and the resulting components tend to exhibit higher defect densities."

Erm.. that's all we get on the matter. In short, I like what he's trying to do here by backing up his models, he just doesn't convince me in this paper that he's got the right explanation.

That said, he concludes that there is an optimum component size range to achieve low fault density. The range differs across languages (because some languages pack less information in per line so the Miller threshold is reached sooner if you're just counting lines) and across programmers (because Miller's magic number is actually a range from 5-9 pieces of information). He mentions several implications of this that make for testable hypotheses. For instance, "manual [code] inspections would be most effective on components that fit into cache", and "only substantial reuse within the same system will likely improve reliability. Modest reuse ... is likely to make it worse."

Paper: A Critical Look at Quality in Large-Scale Simulations

2009-04-07T15:01:00.009-04:00

D. E. Stevenson, "A critical look at quality in large-scale simulations," Computing in Science & Engineering, vol. 1, no. 3, pp. 53-63, 1999.

Here's a wandering article that has lots of great thoughts but never seems to pull them together tightly enough for me to come up with any solid, unified take-aways. Stevenson sets out to describe and tackle the friction, and danger, created by the differing ideas of simulation quality by management and scientists. He's coming at this problem from the perspective of ASCI (Accelerated Scientific Computing Initiative) science -- that is, the folks charged with "predicting, with confidence, the behaviour of nuclear weapons through comprehensive, science-based simulations." After spending some time discussing the disconnect in the understanding of simulation quality and the resulting problems that this disconnect creates, Stevenson takes a step back and looks at the the general modelling and simulation endeavour itself. He explores what modelling and simulation is, why we do it and what we should hope to gain from doing it, and how these two things ought to inform our notion of quality. He also provides us with some observations on why building high-quality simulations is probably difficult, discusses what validation and verification is, and distinguishes between two types of quality. Phew.

Here's Stevenson's summary of the article:

"Software engineering is meant to produce software by a manufacturing paradigm, but this paradigm simply cannot deal with the scientific issues. This article examines the successes and failures of software engineering. I conclude that process does not develop software, people and their tools do. Second, software metrics are not meaningful when the software's purpose is to guarantee the world's safety in the nuclear era. Finally, the quality of simulations must be based on the quality of insights gained from the revealed science."

Okay, some key points to mention. On the topic of what modelling and V&V is, Stevenson introduces some clear terminology. He defines validation and verification in terms of three different types of systems: observational (the world out there -- e.g. the climate), theoretical (our model/theory of the workings of the world -- e.g. the equations that describe climate processes), and calculational (e.g. the implementation of the theoretical model -- the climate model code). Validation checks that the theoretical system properly explains the observational system, and verification checks that the calculational system correctly implements the theoretical system. Stevenson then uses the term validation in a broader sense, stating that "complete validation of the observational-theoretical-calculational systems requires that we compute the right numbers for the right reasons."

On the nature of quality, Stevenson points out that our reasons for modelling need to inform our notions of quality; and that the divide between management and science occurs because this isn't happening. We model, and validate those models, in order to gain insight into the nature of whatever it is that we're modelling (i.e. the observational system). Insight is the essential purpose of science, and simulations are just tools to gain insight. Insight and modelling are the products of science. But, from a manufacturing perspective (read: a management perspective), insight isn't essential for building a model and a simulation. A model can just be seen as a specification (not as a product itself), and a simulation as a final product. Thus, from an from an engineering management position, validation and insight take a back seat -- the real problem is one of manufacturing. And so scientists and management are looking at the same process at cross-purposes.

If insight is the end goal of simulation computing then the quality of computing can be measured by the quality of insight. Since insight leads to knowledge, we can judge the quality of insight by the quality of knowledge we get from a project. How do we judge the quality of knowledge we get? Well... frankly, this is were I lose track of the article a bit. Either it's because I'm just dense, Stevenson is intentionally vague, and/or he's put the real content in another paper of his, D. E. Stevenson, "Science, computational science, and computer science: at a crossroads," in CSC '93: Proceedings of the 1993 ACM conference on Computer science. New York, NY, USA: ACM Press, 1993, pp. 7-14.

What he does say is that whilst we know what scientific and mathematical knowledge looks like (inductive and deductive, respectively), we don't really know what knowledge from computer science looks like. He references a few "principles" of computing knowledge from paper I just mentioned: physical exactness (elimination of parameterisations), computability, and bounded errors (a priori or postepriori error estimates). I'll have to read that paper before I can say much more about that...

Stevenson goes on to describe two kinds of quality, intrinsic and internal. I have to say I'm not quite sure I understand the distinction very well. Here's my take. Stevenson defines intrinsic quality as "the sum total of our faith in the system of models and machines." He says about internal quality, "each dimension [of insight and knowledge we receive from the simulation?], such as the mathematics or the physics, has its own idea of internal quality."

I think what he's doing here is making a distinction in quality that's analogous to the distinction between verification and validation in that intrinsic quality applies to the match between observational and theoretical systems, and internal quality applies to theoretical and calculational systems. Intrinsic quality is an epistemological notion of a good modelling endeavour. It is what we're talking about when we ask: regardless of any possible implementation, what needs to be present in any model and implementation for the modelling effort to be a good one in terms of getting us insight and knowledge?

Internal quality looks at the quality issue from the other side. It assumes (or disregards) intrinsic quality, and focuses just on how good our model and implementation is in terms of the kinds of knowledge we have already. For a mathematician or scientist in general, internal quality may relate to the simplicity or elegance of the model. For a computer scientist or engineer, internal quality may relate to the simplicity or robustness of the code.

Stevenson's point is, I think, that ultimately we computer scientists don't have a clear justification for our measures of quality. If insight and knowledge is the end goal of modelling, we need to have a clear sense of intrinsic quality in our endeavour. Then we need to use this understanding to inform our measures of internal quality. Otherwise we're just measuring things because we can, not because they show us the way to better science.

Paper: Predicting Defects for Eclipse

2009-04-07T11:53:00.005-04:00

T. Zimmermann, R. Premraj, and A. Zeller, "Predicting defects for eclipse," in PROMISE '07: Proceedings of the Third International Workshop on Predictor Models in Software Engineering. Washington, DC, USA: IEEE Computer Society, 2007, pp. 9+.

In this study, Zimmermann et al. map defects found in the Eclipse bug database to the source code, for both pre- and post-release defects. They also calculate several complexity metrics for each file and package, and then explore how those metrics correlate with pre-release defects and post-release defect counts, and briefly how they can be used to predict defect proneness. All of their data is published here. Among other things, their results show a strong correlation between pre- and post-release defects (a buggy package/file is still buggy after release); all complexity measures are at least positively correlated with pre- and post-defect rates (a more complex package/file has more defects); it's possible to learn (linear regression-wise) reasonable models to assess defect proneness for later releases by looking only at a single release.

What's interesting to me about this study is the definition of defect used here, and the method of counting defects. Zimmerman et al. define a defect by the bug report and the associated code change that fixes it. In this way, a defect is defined as anything worth fixing. In Hatton's terms, this definition covers both faults and failures, but limits it to only those problems the users and developers find relevant.

Programmatically counting defects is done in two steps: in the first step fixes are identified by searching through the version control change log for entries that contain references to bugs (e.g. "'fixed 42233' or 'bug #23444'"); in the second step the release the fix applies to is determined by looking at the bug report in the bug tracking system. This method could be adapted to any project where the developers consistently mark fixes with a reference to the bug tracking system or release number (including posting comments in the code). The authors reference three other papers which use a similar technique.

Paper: The T Experiments

2009-04-06T09:26:00.006-04:00

L. Hatton, "The t experiments: errors in scientific software," Computational Science & Engineering, IEEE, vol. 4, no. 2, pp. 27-38, 1997.

Back in 1997, Les Hatton published a study composed of two experiments to test scientific software quality. The first, which he calls T1, involved the static analysis of over 100 pieces of scientific software from a broad range of application areas. The second experiment, T2, involved dynamic analysis of 9 different seismic data processing programs, each one supposedly designed to do the same thing. In a nutshell, the results suggest that scientific software has plenty of statically detectable faults (and the number of faults varied widely across the different programs analysed), and that there is significant and unexpected uncertainty in the output of this software -- agreement amongst the seismic processing packages is only to 1 significant digit. Hatton says, "Taken with other evidence, the T experiments suggest that the results of scientific calculations carried out by many software packages should be treated with the same measure of disbelief researchers have traditionally attached to the results of unconfirmed physical experiments."

Hatton makes the distinction in the paper between two types of defects: faults and failures. A fault is a statically detectable trouble area in the software code; "a misuse of the language which will very likely cause the program to fail in some context". A failure is a defect measured at runtime (for example, a program crash or an output inaccuracy). There is a uncertainty associated with software faults -- for many of them (say for example, assigning a pointer to an integer variable) it's not sure bet that they will "mature" into a runtime failure. Hatton weights each kind of fault with a severity rating which amounts to the likelihood the fault could cause a failure, a "rough risk factor". It's these weighted fault rates he publishes in the paper. He doesn't go into detail about the ratings, so I can't comment other than to say I'm already suspicious of something so subjective being used this way. More details of the T1 experiment are available in his book, L. Hatton, Safer C: Developing Software for in High-Integrity and Safety-Critical Systems. New York, NY, USA: McGraw-Hill, Inc., 1995.

I'd like to note that what Hatton is doing here in this study is looking at various bits of software on his own, at a single point in time, in order to objectively determine code quality. In my pitch about investigating climate modelling code quality I was thinking more about looking at the defects uncovered by the software users and developers themselves. The defects identified[1] by the climate scientists will contain both faults and failures. The faults found by scientists are sure to be relevant and important whereas the same is probably not true for all of the "generic" faults detected by static analysis software. Either way, both of these methods of gauging quality are incomplete in some sense: both use heuristics to find defects (if you consider the testing scientists do as a "heuristic").

This paper provides some partial statistics I could use to compare with climate models. Hatton used static analysis software from Programming Research Ltd., so it's at least feasible I could run the climate models through the same analysis. I'm not sure what the value would be for doing that just yet. Partly, because it's not clear what I'd be comparing against: Hatton analyses software from "40 application areas, including, for example, graphics, nuclear engineering, mechanical engineering, chemical engineering, civil engineering, communications, databases, medical systems, and aerospace."

[1] I suspect defects in one version to show up over the course developing later versions, but some defects may be known ahead of time. I'm not sure about how to count these yet.

On Climate Model Software Quality

2009-03-30T17:07:00.004-04:00

I've abandoned the search for a thesis surrounding an education tool for climate change. It just wasn't getting clearer where to go with it or how to pitch it from a software engineering angle.

So here's a new topic. Whilst spending last summer at the Hadley Centre, Steve made the preliminary observation that the defect density of Hadley Centre's climate model appears to be surprisingly lower than defect rates for comparably-sized projects. Does this observation hold up under scrutiny? What if we control carefully for project size and user and developer base size? If we were to compare the kinds of defects found in the other projects to those found in the Hadley GCM, surely we'd find that there are classes of defects that are rarely considered defects by the Hadley scientists (e.g. superficial bugs, like GUI defects, for instance). So what exactly do scientists consider as defects? Can these be characterised? If we only compared the defect density between projects over similar classes of defects, do we still see the lower defect rate in the climate model? How do other GCMs or climate models compare?

Regardless of the outcome of a more rigorous look at climate model defects, there are larger questions of software quality to explore. Namely, what is the underlying cause for the differences in defect density (whether the GCM defect density is, after all, better, worse, or comparable)? If the defect density is lower for climate models, one hypothesis may be that it's a result of the fact that climate scientists are both the users and developers of their software, so maybe they are more likely to catch defects early on. But then, we'd expect to see similar defect density patterns in open source software. Another hypothesis is that climate models are inherently more "resistant" to defects because of the powerful constraints put on them by the physical systems they simulate (e.g. conservation of mass, and energy) and the extreme numerical sensitivity of the models. Or maybe the folks at Hadley have a great software engineering process that others need to learn from.

Thoughts? What am I missing?

There is a mountain of literature on the nature of scientific software quality, defect density, and related topics. I'm just starting into it now. Here's a glimpse into where I am:

L. Hatton, "The t experiments: errors in scientific software," Computational Science & Engineering, IEEE, vol. 4, no. 2, pp. 27-38, 1997.

My feed of interesting articles

2009-03-13T17:09:00.002-04:00

Google Reader has a great feature whereby you can share articles from news feeds you find interesting. Here's a link to my shared items page, which you can subscribe to:

http://www.google.com/reader/shared/15421741800551159855

Of course, I could just create a blog post about the item and link to it, but this is much much easier and keeps my blog as a place to post (slightly!) more developed thoughts.

Climate Interactive goes to Copenhagen

2009-03-12T12:36:00.003-04:00

In December of 2009, the Kyoto Protocol gets revisited and remade at the Climate Conference in Copenhagen. Leading up to that conference is the scientific Climate Congress in Copenhagen, which is happening right now (in fact, today is the last day). This conference aims to synthesise "existing and emerging scientific knowledge necessary in order to make intelligent societal decisions concerning application of mitigation and adaptation strategies in response to climate change."

The folks from Climate Interactive (see also) are there and gave a presentation about their climate simulation software, C-ROADS, that's designed especially for use as a decision making tool for policy makers. It's fast, simple to understand and use, and produces predictions inline with the accepted climate science. The great thing is that it's being used -- read their blog for lots of examples (in particular, see the recent post about John Kerry using C-ROADS).

In two of the extra slides of the Climate Interactive presentation, they state:

It is difficult for decision makers to:
aggregate diverse emissions reductions proposals into a single global emissions projection and
mentally simulate from that emissions projection the resulting atmospheric CO2 level or temperature increase.
Tools are needed to help decision makers assess whether policy options are sufficient to achieve goals for stabilizing CO2 levels and limiting global temperature increase to within a safe range.

This is quite similar to the motivation behind the educational tool idea I pitched a while back. Hrm..

Structured brainstorming exercise

2009-03-11T18:59:00.002-04:00

Steve and I have been talking about how to generate ideas on how software engineering research can be applied to the climate crisis.

Here's an idea for a structured brainstorming exercise that's inspired by Edward de Bono's Six Thinking Hats technique. Instead of the thinking hats that de Bono suggests we instead have a hat for each software engineering sub-discipline (e.g. requirements analysis, software design, formal methods, software testing/quality, etc..). The brainstorming is done with participants "wearing" only one of these Research Hats at a time; while wearing a hat the participant focuses on thinking about applying only their hat's research area to a given problem.

We gather a list of climate change issues, and get participants to brainstorm research problems that come out of each issue applicable to their research area.

Using research area Hats is purely a technique to focus the brainstorming conversations, so that they stay within software engineering and so that we make sure to get a variety of viewpoints on each issue. I imagine the exercise carried out in teams, with each team taking turns all wearing the same hat, and flipping through the various issues. It also makes sense to dispense with the research hat idea if the participants are experts in one particular sub-discipline, in which case the exercise is simply, "How can your research, or research discipline, be applied to this issue?"

Our weekly climate change chats have explored some rough ideas of the issues. See these posts for my thematic summary of our discussions.

Climate policy engineering

2009-03-08T16:11:00.003-04:00

Attention: wild speculation and provocative operations below.

Steve and I met last week and discussed a rather intruiging, and far out topic: the relationship between software design and global climate policy design. I'll lay it out rather straightforwardly and unconditionally:

Software systems are some of the most complex systems that humans design. Taking a global view, the world's climate policies also form a complex system of human design. Are there any design or process techniques from software engineering that can be applied to climate policy design?

(In short, just "s/software/climate policy/g" and see what makes sense.)

It doesn't seem like climate policy is built nearly as deliberately or methodically as software often is. You might say that it's "hacked". Can insight from software development processes be used to guide the process of planetary climate policy development?

Software designers use an architecture to cope with the complexity of large software systems (btw, is this a form of chunking?). Climate policy (policy in general?) lacks this sort of architecture (or maybe lacks a good one?). What can we use from the design of software architectures to help us design good climate policy? Are there design patterns in effective climate policy that match up with the software engineer's idea of software design patterns?

You can run the analogy along all the different aspects of software engineering: requirements engineering, design, testing, quality, development process, etc...

Does the analogy work? Also, can we go the other way around and import useful concepts from the way climate policy is designed into software engineering? Ugh.. ;-)

I suppose what we're doing here is applying systems engineering thinking to climate policy design by way of our knowledge of software engineering (as one type of systems engineering).

By the way, Steve has just blogged a much more coherent statement of how software engineering can play a role in fighting climate change.

Lakoff, Systems Thinking, Obama, Sweeney

2009-03-05T16:55:00.002-05:00

Check out Linda Sweeney's recent blog post on Systems Thinking + The Obama Code. She elaborates on one of the seven characteristics that George Lakoff identifies as part of the "Obama Code": "seven crucial intellectual moves [by Obama] that [Lakoff believes] are historically, practically, and cognitively appropriate, as well as politically astute."

CO2 emissions data on Google Earth

2009-02-20T15:32:00.002-05:00

Tweet. A Google Earth for displaying CO2 emission info across North America. See this WSJ article about it -- includes a video demo. Very cool.

(via nature.com)

Another SE/Climate Change theme: decision support

2009-02-16T18:44:00.005-05:00

(Note: I've tagged all the previous posts on software engineering and climate change with the label "SECC". Read them here, or get the feed here.)

Another theme in the software engineering and climate change arena would have to be that of decision support. We cover this somewhat in our discussions of raising awareness (where, for instance, we talked about a handheld tool that reads out the supply chain carbon footprint of a produce in the supermarket) but I think somehow it's helpful to separate the two themes. Awareness raising includes much more passive and less specific activities than decision making support tools.

Erm, but that's just preamble, really. Mainly I just wanted to post this example of a decision support tool for climate change: MIT Centre for Collective Intelligence's Climate Collaboratorium.

This system, currently under development, will use an innovative combination of internet-mediated interaction, collectively generated idea repositories, computer simulation, and explicit representation of argumentation to help large, diverse, and geographically-dispersed groups systematically explore, evaluate, and come to decisions concerning systemic challenges.

For something less wordy take a look at the YouTube video on the "deliberation component" is a little clearer. It looks like a set of protocols for using an online discussion board in the capacity of an argument map.

Al Gore tells scientists to speak up

2009-02-16T18:27:00.002-05:00

Al Gore tells the American Association for the Advancement of Science that scientists need to speak up about the urgency of climate change:

"... the U.S. coal industry has invested a half-billion dollars in advertising and lobbying to reinforce the misleading message that coal is clean and efficient.

"When they spend $500 million putting their version of this story in the minds of the American people," he told the AAAS audience, "it increases the importance of you being willing to speak out and, as civic scientists, of finding ways to communicate the truth about what this huge increase in global warming pollution is doing.

... [big snip here] ...

"If I could," Gore concluded, "I would motivate you to leave this city after this meeting and start getting involved in politics. Keep your day job, but start getting involved in this historic debate. We need you."

Read more about it here (includes video).

via Climate Feedback on Nature.com (which, curiously, references Gore's speech with a link to a Google News search).