In Search of a Paradigm: The Science of Methods

Regular readers of the Phaneron may consider this essay as a departure from the usual subjects discussed here, those being of philosophy, logic, semiotics and singularity. However, the astute reader will recognize the close association between the subject of this essay and the philosophy of science.

2012 was a year where I took the opportunity to focus on methods. Methods have been an interest of mine for most of my career. This interest lead to my certification in the Rational Unified Process, SCRUM Master and the Software Engineering Institute’s CMMI-DEV. Late in 2011 I became active in the Theory Track of Ivar Jacobson’s Software Engineering Method and Theory (SEMAT) initiative. My work on the Theory Track motivated me to engage the US Federal CIO Council’s Management Best Practices Committee to establish an Agile Practices Working Group and to become an affiliate in the Software Engineering Institute‘s Affiliate Program. Although my motivations were practical, I believe the result of this year’s focus is more than that. I believe the result is significant. What’s significant is that I can now articulate a problem statement that has perplexed me and I suspect many of us in software for the past few decades. In the rest of this essay, I will begin with a situation analysis derived from a few informal Theory Track discussions, then explain the problem statement. Following from the problem statement, I’ll propose the hypothesis that software engineering doesn’t need another method, it needs a science of methods, then I will describe how we as software engineers should analyze and reason about methods.

Before I begin, I want to thank Pontus Johnson for the thoughtful email discussion. I also want to thank Ivar Jacobson for his leadership in creating the environment where a such a discussion can take place. There are few opportunities to have this type of discussion. The workplace is generally not one. So many thanks to Ivar. For those of us lucky enough to work with him, his selfless leadership is as much a part of his legacy as his many accomplishments.

I’ll begin by framing our situation analysis in terms of the following questions: Have you ever wondered why software engineering has so many methods? And have you wondered why none of the proposed methods can be conclusively shown to improve, serve as an complete explanation for and consistently avoid regularly recurring failures on software projects? Do you find it unsatisfactory that our profession encourages certain practitioners to propose a method while at the same time hold that this same method cannot be defined or its practices independently verified? Have you questioned whether what we as software professionals really face is a trade off between a metaphorical cathedral and a bazaar? Have you ever thought that you’re not really looking for a silver bullet, just a known approach you can recommend to a senior executive without later finding both unexplainable cost overruns and collateral damage to your reputation? Is it that we just haven’t found the right method and that one’s just around the corner? If you’ve asked yourself these questions, then you’ve asked yourself something fundamental about the state of our profession.

It was 1970 when Winston Royce published Managing the Development of Large Software Systems. Although new methods have been proposed since, there’s little evidence that any of the methods proposed are responsive to the questions posed in our situation analysis. No doubt we’ve had the time. I count 42 years since 1970. And we’ve engaged in a certain kind of institutional research – the Software Engineering Institute has been around for 25+ years – as well as non-institutional approaches which claim to be based on empiricism or meritocracy. The results are underwhelming. Failure of software projects is still too much the norm, especially here in Washington, DC. And both high cost and risk still too often characterize the environment in which we work. In short, we’re in a do loop and just to develop a problem statement from our situation analysis, we need a different perspective if we expect a different result than what we have so far.

Thomas Kuhn’s Structure of Scientific Revolutions was required reading in my high school and I’ve had a copy in my bookshelf ever since. Very early in the development of SEMAT, Ivar Jacobson posed the question: Where are Maxwell’s Equations for Software Engineering? I believe this an excellent question and one that with some refinement helps frame the problem statement following from our situation analysis. Ivar’s question also implicitly formulates the hypothesis that a paradigm can, but does not yet exist for software engineering. I have challenged this hypothesis in the Theory Track discussions. Research on the legitimacy of paradigms in the social sciences remains inconclusive. The interested reader will enjoy the Preface to the Second Edition and its Postscript as well as the Stanford Encyclopedia of Philosophy article on Scientific Revolutions.

Distinguished sociologist Charles Tilley pessimistically says here that “The social sciences have never developed as connected a body of knowledge [as the natural sciences], integrated with a set of procedures for verifying that knowledge and the social sciences have remained a contested terrain and that’s something people often try to articulate incorrectly …” Kuhn chooses to write exclusively about the natural sciences in the Structure of Scientific Revolutions. In the chapter called the Route to Normal Science, Kuhn says that it is an open question whether “parts of social science have yet acquired such paradigms at all.” Unfortunately, Kuhn does not elaborate on what parts of the social sciences may or may not have formulated paradigms. If even Kuhn leaves doubt whether social sciences have formulated paradigms, then what of software engineering? It is unlikely that we will find the natural laws of software engineering. And it would be optimistic to conclude that software engineering as we practice it today is a social science. To equate software engineering with the natural sciences would both be an error and set an indirect course for a science of methods.

Our problem statement then is this: If we cannot expect to formulate a paradigm for software engineering as one might expect from a natural science, or more optimistically a social science, must it remain nothing more than social engineering? If we require more than social engineering in response to our situation analysis, is a paradigm necessary for a science of methods? If a paradigm is not necessary, what is a science of methods and what are its minimal parts?

I believe many, if not all, software engineers would endorse the belief that software engineering needs more than social engineering in response to our situation analysis. There are no shortage of people who are not software engineers who would not. I will count general managers who lack experience in software who believe they can manage anything, venture capitalists who believe they can slide a pizza box under a locked door and have a multi-million dollar application appear just in time to execute an exit strategy, those who believe they are the next Mark Zuckerberg, and quick and dirty civic hackers who believe they can compete on a $100 million project, to name just a few. Those who believe software engineering should remain nothing more than social engineering require us all to accept less from ourselves than what we might expect from a physician, lawyer, accountant or plumber. We expect more than social engineering from others and we should expect it from ourselves.

Good news. A paradigm is not necessary for a science of methods. Those who read the Scientific Revolutions article in the Stanford Encyclopedia of Philosophy know there are competing theories of scientific analysis. This comment is not to minimize Kuhn’s work, but to recognize diversity in the history and philosophy of science as well as remove the dependency on only one of those theories in developing a science of methods.

So what is a science of methods? Simply put, a science of methods is the application of the scientific method to software engineering. That’s a pretty simple definition and I’ll just say these three things about it. 1) With all this talk about science I do not mean to minimize the social aspects of teaming. I mean that without science, the social aspects are not sufficient. 2) When we talk about a theory of software engineering on the SEMAT Theory Track we separate software engineering from computer science as two distinct subject areas. 3) We do not expect that the science of methods will find the missing method, the method of all methods, or the meta-method. There’s no one method, just like there’s no one ring. I want to take the opportunity to endorse the good work of SEMAT, both its proposal that methods are a composition of practices and the development of a kernel of practices, from which methods can be composed. I believe this work is consistent with a science of methods. I also think the natural follow-on to the current SEMAT activities is how to analyze and reason about methods composed from practices in the kernel.

Analyzing methods and reasoning about methods are the minimal parts of the science of methods. I will mention a few other desirable parts below. I make no attempt to provide an ontology of the science of methods. The effort to bound that analysis might be longer than the current essay. I will limit the analysis of methods to the qualities of methods that allow us to judge their nature whether having to set them in relation to, or remain independent of something else. It is not that a method is invalid if it fails to satisfy these qualities. More importantly we recognize the method for those qualities it possesses rather than avoid acknowledging the strengths and weaknesses of a method. The qualities also help us in composing practices from the kernel. So, the qualities are as follows:

  1. Sound. A method is sound if it is not inconsistent. Inconsistent means the method contains contradictory statements.
  2. Complete. A method is complete if it is not missing something. We leave the boundary problem undefined.
  3. Intentional. A method that is intentional has and is aligned with a purpose. We leave the purpose undefined.
  4. Parsimonious. A method that is parsimonious has no extra statements. We allow for synonyms in the terms.
  5. Falsifiable. A method that contains statements that are not falsifiable – true or false – is ambiguous. This is SEMAT Principle #6.
  6. Objective. A method that is objective both contains only statements whose criteria for falsifiability and method of reasoning are defined. This is a little more than SEMAT principle #12.

I define reasoning about methods in terms of reaching a conclusion based on premises in the usual way one would expect from logic. The science of methods allows for the already well known subjects of deduction, induction and abduction. While allowing these broad categories of logics in reasoning about methods, we also require a statement of the proof theory under which the conclusion was reach from the premises. For those not familiar with abduction, abduction is evidence that a is necessary but not sufficient to conclude b. The science of methods also allows the weaker notion of indication by John W. Tukey. Tukey’s observations on indication, determination and inference provide a practical approach to reason under uncertainty. When we reason about a method using indication, we expect the acknowledgement of which proof theory and its provenance allowed reaching the conclusion.

With the minimal parts of the science of methods defined, I remain pragmatic and acknowledge that experimentation is a desired part of the science of methods to be defined later. No doubt industrial research promises empirical results from experimentation, but isolating controlled variables in a industrial setting remains a significant challenge in most if not all research into software engineering practices. There may also be other desired parts later defined.

Some 42 years after Winston Royce published Managing the Development of Large Software Systems we are still in search of a paradigm. It’s unlikely we’ll find one like the natural sciences, but with the excellent foundation provided by SEMAT I have defined here the minimal parts of a science of methods. The science of methods does not preclude social engineering in software engineering. We simply ask from ourselves, what we’d ask from others.

In later posts I will describe my recent experiences with the Software Engineering Institute and the CIO Council examining “agile practices” for the US Federal government. I will apply the science of methods to claims about methods in general and analyze reports and guidance from certain unidentifed “state actors.”