Tag Archives: computation

Searle’s wall, computation and representation

Posted on by

Reading a bit more of Brain Cantwell Smith’s “On the Origin of Objects”  and he refers (p.30-31) to Searle‘s wall that, according to Searle, can be interpreted as implementing a word processor.  This all hinges on predicates introduced by Goodman such as ‘grue’, meaning “green is examined before time t or blue if examined after”:

grue(x) = if ( now() < t ) green(x)
          else blue(x)

The problem is that an emerald apparently changes state from grue to not grue at time t, without any work being done.  Searle’s wall is just an extrapolation of this so that you can interpret the state of the wall at a time to be something arbitrarily complex, but without it ever changing at all.

This issue of the fundamental nature of computation has long seemed to me the ‘black hole’ at the heart of our discipline (I’ve alluded to this before in “What is Computing?“).  Arguably we don’t understand information much either, but at least we can measure it – we have a unit, the bit; but with computation we cannot even measure except without reference to specific implementation architecture whether Turing machine or Intel Core.  Common sense (or at least programmer’s common sense) tells us that any given computational device has only so much computational ‘power’ and that any problem has a minimum amount of computational effort needed to solve it, but we find it hard to quantify precisely.  However,  by Searle’s argument we can do arbitrary amounts of computation with a brick wall.

For me, a defining moment came about 10 years ago, I recall I was in Loughbrough for an examiner’s meeting and clearly looking through MSc scripts had lost it’s thrill as I was daydreaming about computation (as one does).  I was thinking about the relationship between computation and representation and in particular the fast (I think fastest) way to do multiplication of very large numbers, the Schönhage–Strassen algorithm.

If you’ve not come across this, the algorithm hinges on the fact that multiplication is a form of convolution (sum of a[i] * b[n-i]) and a Fourier transform converts convolution into pointwise multiplication  (simply a[i] * b[i]). The algorithm looks something like:

1. represent numbers, a and b, in base B (for suitable B)
2. perform FFT in a and b to give af and bf
3. perform pointwise multiplication on af and bf to give cf
4. perform inverse FFT on cf to give cfi
5. tidy up cfi a but doing carries etc. to give c
6. c is the answer (a*b) in base B

In this the heart of the computation is the pointwise multiplication at step 3, this is what ‘makes it’ multiplication.  However, this is a particularly extreme case where the change of representation (steps 2 and 4) makes the computation easier. What had been a quadratic O(N2) convolution is now a linear O(N) number of pointwise multiplications (strictly O(n) where n = N/log(B) ). This change of representation is in fact so extreme, that now the ‘real work’ of the algorithm in step 3 takes significantly less time (O(n) multiplications) compared to the change in representation at steps 2 and 4 (FFT is O( n log(n) ) multiplications).

Forgetting the mathematics this means the majority of the computational time in doing this multiplication is taken up by the change of representation.

In fact, if the data had been presented for multiplication already in FFT form and result expected in FFT representation, then the computational ‘cost’ of multiplication would have been linear … or to be even more extreme if instead of ‘representing’ two numbers as a and b we instead ‘represent’ them as a*b and a/b, then multiplication is free.  In general, computation lies as much in the complexity of putting something into a representation as it is in the manipulation of it once it is represented.  Computation is change of representation.

In a letter to CACM in 1966 Knuth said1:

When a scientist conducts an experiment in which he is measuring the value of some quantity, we have four things present, each of which is often called “information”: (a) The true value of the quantity being measured; (b) the approximation to this true value that is actually obtained by the measuring device; (c) the representation of the value (b) in some formal language; and (d) the concepts learned by the scientist from his study of the measurements. It would seem that the word “data” would be most appropriately applied to (c), and the word “information” when used in a technical sense should be further qualified by stating what kind of information is meant.

In these terms problems are about information, whereas algorithms are operating on data … but the ‘cost’ of computation has to also include the cost of turning information into data and back again.

Back to Searle’s wall and the Goodman’s emerald.  The emerald ‘changes’ state from grue to not grue with no cost or work, but in order to ask the question “is this emerald grue?” the answer will involve computation (if (now()<t) …).  Similarly if we have rules like this, but so complicated that Searle’s wall ‘implements’ a word processor, that is fine, but in order to work out what is on the word processor ‘screen’ based on the observation of the (unchanging) wall, the computation involved in making that observation would be equivalent to running the word processor.

At a theoretical computation level this reminds us that when we look at the computation in a Turing machine, vs. an Intel processor or lambda calculus, we need to consider the costs of change of representations between them.  And at a practical level, we all know that 90% of the complexity of any program is in the I/O.

  1. Donald Knuth, “Algorithm and Program; Information and Data”, Letters to the editor. Commun. ACM 9, 9, Sep. 1966, 653-654. DOI= http://doi.acm.org/10.1145/365813.858374 [back]

reading: Computing is a Natural Science

Posted on by

I’ve just been re-reading Peter Denning’s article “Computing is a Natural Science”1.    The basic thesis is that computation as broad concept goes way beyond digital computers and many aspects of science and life have a computational flavour; “Computing is the study of natural and artificial information processes“.  As an article this is in some ways not so controversial as computational analogies have been used in cognitive science for almost as long as there have been computers (and before that analogies with steam engines) and readers of popular science magazines can’t have missed the physicists using parallels with information and computation in cosmology.  However, Denning’s paper is to some extent a manifesto, or call to arms: “computing is not just a load of old transistor, but something to be proud of, the roots of understanding the universe”.

Particularly useful are the “principles framework for computing” that Denning has been constructing through a community discussion process.  I mentioned these before when blogging about “what is computing? The article lists the top level categories and gives examples of how each of these can be found in areas that one would not generally think of as ‘computing’.

Computation (meaning and limits of computation)
Communication (reliable data transmission)
Coordination (cooperation among networked entities)
Recollection (storage and retrieval of information)
Automation (meaning and limits of automation)
Evaluation (performance prediction and capacity planning)
Design (building reliable software systems)
 categories from “Great Principles of Computing“

Occasionally the mappings are stretched … I’m not convinced that natural fractals are about hierarchical aggregation … but they do paint a picture of ubiquitous parallels across the key areas of computation.

Denning is presenting a brief manifesto not a treatise, so examples of very different kinds tend to be presented together.  There seem to be three main kinds:

  • ubiquity of computational devices – from iPods to the internet, computation is part of day-to-day life
  • ubiquity of computation as a tool –  from physical simulations to mathematical proofs and eScience
  • ubiquity of computation as an explanatory framework – modelling physical and biological systems as if they were performing a computational function

It is the last, computation as analogy or theoretical lens that seems most critical to the argument as the others are really about computers in the traditional sense and few would argue that computers (rather than computation) are everywhere.

Rather like weak AI, one can look treat these analogies as simply that, rather like the fact that electrical flow in circuits can be compared with water flow in pipes.  So we may feel that computation may be a good way to understand a phenomena, but with no intention of saying the phenomena is fundamentally computational.

However, some things are closer to a ‘strong’ view of computation as natural science.  One example of this is the “socio-organisational Church-Turing hypothesis”, a term I often use in talks with its origins in a 1998 paper “Redefining Organisational Memory“.  The idea is that organisations are, amongst other things, information processing systems, and therefore it is reasonable to expect to see structural similarities between phenomena in organizations and those in other information processing systems such as digital computers or our brains.  Here computation is not simply an analogy or useful model, the similarity is because there is a deep rooted relationship – an organisation is not just like a computer, it is actually computational.

Apparently absent are examples of where the methods of algorithmics or software engineering are being applied in other domains; what has become known as ‘computational thinking‘. This maybe because there are two sides to computing:

  • computing (as a natural science) understanding how computers (and similar things) behave – related to issues such as emergence, interaction and communication, limits of computability
  • computing (as a design discipline) making computers (and similar things) behave as we want them to – related to issues such as hierarchical decomposition and separation of concerns

The first can be seen as about understanding complexity and the latter controlling it.  Denning is principally addressing the former, whereas computational thinking is about the latter.

Denning’s thesis could be summarised as “computation is about more than computers”.  However, that is perhaps obvious in that the early study of computation by Church and Turing was before the advent of digital computers; indeed Church was primarily addressing what could be computed by mathematicians not machines!  Certainly I was left wondering what exactly was ubiquitous: computation, mathematics, cybernetics?

Denning notes how the pursuit of cybernetics as an all embracing science ran aground as it seemed to claim too much territory (although the practical application of control theory is still live and well) … and one wonders if the same could happen for Denning’s vision.  Certainly embodied mind theorists would find more in common with cybernetics than much of computing and mathematicians are not going to give up the language of God too easily.

Given my interest in all three subjects, computation, mathematics, cybernetics, it prompts me to look for the distinctions, and overlaps, and maybe the way that they are all linked (perhaps all just part of mathematics ;-)).  Furthermore as I am also interested in understanding the nature of computation I wonder whether looking at how natural things are ‘like’ computation may be not only an application of computational understanding, but more a lens or mirror that helps us see computation itself more clearly.

  1. well when I say re-reading I think the first ‘read’ was more of a skim, great thing about being on sabbatical is I actually DO get to read things 🙂 [back]

What is Computing? The centrality of systemics

Posted on by

Recently I was in a meeting where the issue of ‘core’ computer science came up. One person listed a few areas, but then this was challenged by another member of the group who said (to be fair, partly in jest), that core computer science should certainly include computer architecture, but not the ‘human stuff’.

I felt a little like a teenager complete with T-shirt and iPod dropped into Jurassic Park arguments that I thought had been put to bed in the 1980s suddenly resurfacing – how do you explain this white thing that makes sounds from its earphones to a caveman wearing skins?

However, I also felt a certain sympathy as I often wonder about computer science as a whole; indeed it has its own arguments in the 1960s and ’70s as to whether it was a ‘discipline’ as opposed to just an application domain for maths or electronics, or just a tool for business. Maybe one of the clinchers was the theoretical foundations of computing in the work of Church and Turing … but strangely enough at Lancaster the closest to this, the course on algorithmic complexity, is taught by a HCI person!

One of my worries in computing is that these theoretical foundations are still weak, there is black hole in the theoretical centre of computer science1. However, these theoretical issues were certainly not what was bothering my colleague. To answer his challenge and my own worries about the discipline we really need to know – what is computing?

Continue reading →

  1. This demands a discussion of its own, but the basic problem is that while Church and Turing gave us understanding of disembodied computation, we still do not have clear understanding of generic computation when embodied in devices in general only particular architectures. [back]