Alan Dix > 2010

From the Web of Data to a World of Action

Alan Dix1,2 , Giorgos Lepouras3, Akrivi Katifori4, Costas Vassilakis4, Tiziana Catarci5, Antonella Poggi5, Yannis Ioannidis4, Miguel Mora6, Ilias Daradimos3,4, Nazihah Md.Akim1, Shah Rukh Humayoun5, Fabio Terella7

1 Computing Department, Lancaster University, Lancaster, UK
2 Talis, Birmingham, UK
3 Dept. of Computer Science and Technology, University of Peloponnese, Tripolis, Hellas (Greece)
4 Department of Informatics & Telecommunications, University of Athens, Athens, Hellas (Greece)
5 Dipartimento di Informatica e Sistemistica, Universita' di Roma "La Sapienza", Rome, Italy
6 Escuela Politécnica Superior,Universidad Autonoma de Madrid, Madrid, Spain
7 EXALTECH S.r.l., Rome, Italy

Published in Journal of Web Semantics, Special Issue on User Interaction in Semantic Web research. Vol. 8, Iss. 4, Nov. 2010, pp. 394-408. doi:10.1016/j.websem.2010.04.007

Download draft paper (PDF, 872K)

Abstract

This paper takes as its premise that the web is a place of action, not just information, and that the purpose of global data is to serve human needs.  The paper presents several component technologies, which together work towards a vision where many small micro-applications can be threaded together using automated assistance to enable a unified and rich interaction.  These technologies include data detector technology to enable any text to become a start point of semantic interaction; annotations for web-based services so that they can link data to potential actions; spreading activation over personal ontologies, to allow modelling of context; algorithms for automatically inferring 'typing' of web-form input data based on previous user inputs; and early work on inferring task structures from action traces.  Some of these have already been integrated within an experimental web-based (extended) bookmarking tool, Snip!t, and a prototype desktop application On Time, and the paper discusses how the components could be more fully, yet more openly, linked in terms of both architecture and interaction.  As well as contributing to the goal of an action and activity-focused web, the work also exposes a number of broader issues, theoretical, practical, social and economic, for the Semantic Web.

keywords: task support, spreading activation,intelligent user interfaces
Full reference:
A. Dix , G. Lepouras, A. Katifori, C. Vassilakis, T. Catarci, A. Poggi, Y. Ioannidis, M. Mora, I. Daradimos, N. Md.Akim, S.R. Humayoun, F. Terella (2010). From the Web of Data to a World of Action. Journal of Web Semantics, Special Issue on Exploring New Interaction Designs Made Possible by the Semantic Web, 8(4), pp.394-408
http://www.hcibook.com/alan/papers/
JWS-2010-world-of-action/
more:
Springer page for final paper
TIM Project web page
play with Snip!t

References

  1. [AS68] C. R. Atkinson, M. R. Shiffrin, Human memory: A proposed system and its control processes. in The psychology of learning and motivation, ed. K.W. Spence and J.T. Spence, vol. 8. (Academic Press, 1968).
  2. [An83] J. R. Anderson, A Spreading Activation Theory of Memory, Journal of Verbal Learning and Verbal Behaviour, 22 (1983) 261-295
  3. [AK97] Ashish, N. and Knoblock, C. A. 1997. Semi-Automatic Wrapper Generation for Internet Information Sources. In Proceedings of the Second IFCIS international Conference on Cooperative information Systems (June 24 - 27, 1997). COOPIS. IEEE Computer Society, Washington, DC, 160-169.
  4. [BB09] BBC Backstage. British Broadcasting Corporation, dated 2004-2005, Accessed 6/7/2009. http://backstage.bbc.co.uk/
  5. [BF92] R. Beale and J. Finlay, Eds. 1992 Neural Networks and Pattern Recognition in Human-Computer Interaction. Ellis Horwood.
  6. [BH10] C. Bizer, T. Heath and T. Berners-Lee (to appear) Linked Data - The Story So Far. International Journal on Semantic Web and Information Systems, Special Issue on Linked Data. Preprint at: http://tomheath.com/papers/bizer-heath-berners-lee-ijswis-linked-data.pdf
  7. [BL07] T. Berners-Lee (2007). Looking Back, Looking forward: The process of designing things in a very large space.  Inaugural Lecture, Southampton University, 14th March 2007. http://www.w3.org/2007/Talks/0313-bcs-tbl/
  8. [BP98] S. Brin, and L. Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine, in Proc. of Seventh ACM International World-Wide Web Conference (WWW ‘98), Brisbane, Australia, 1998
  9. [CD07] T. Catarci, A. Dix, A. Katifori, G. Lepouras and A. Poggi (2007). Task-Centered Information Management. In First International DELOS Conference, Pisa, Italy, February 13-14, 2007, Revised Selected Papers. , C. Thanos, F. Borri and L. Candela (eds.). LNCS 4877, Springer, pp. 197-206.
  10. [CD07] D. Calvanese, G. De Giacomo, D. Lembo, M. Lenzerini, and R. Rosati. Tractable Reasoning and Efficient Query Answering in Description Logics: The DL-Lite family. Journal of Automated Reasoning, volume 39, number 3, pages 385-429, 2007.
  11. [CH94] Carr, L. A., Hall, W., Davis, H. C., DeRoure, D. C. and Hollom, R. (1994) The Microcosm Link Service and its Application to the World Wide Web. In: Proceedings of the First WWW Conference, Geneva..
  12. [CH06] T. Catarci, B. Habegger, and A. Poggi (2006). Intelligent User Task Oriented Systems. Proceedings of the Second SIGIR Workshop on Personal Information Management (PIM), 2006.
  13. [CG08] T. Catarci, R. Giuliano, M. Piva, A. Poggi, F. Terella, E. Tracanna (2008). The On-TIME user interface. In the 5th Conference of the Italian Chapter of AIS (ItAIS 2008), Paris, France, 2008.
  14. [Cl98] Clark, A.: Being There: Putting Brain, Body and the World Together Again. MIT Press, Cambridge, MA (1998)
  15. [CL75] A. M. Collins and E. F. Loftus. A spreading-activation theory of semantic processing. Psychological Review, 82:407–425, 1975.
  16. [CM04] Crescenzi, V. and Mecca, G. 2004. Automatic information extraction from large websites. J. ACM 51, 5 (Sep. 2004), 731-779. DOI= http://doi.acm.org/10.1145/1017460.1017462
  17. [Cr97] F. Crestani, Application of spreading activation techniques in information retrieval, Artificial Intelligence Review 11(6) (1997) 453–482.
  18. [Cy91] Cypher, A. 1991. EAGER: programming repetitive tasks by example. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Reaching Through Technology (New Orleans, Louisiana, United States, April 27 - May 02, 1991). S. P. Robertson, G. M. Olson, and J. S. Olson, Eds. CHI '91. ACM, New York, NY, 33-39. DOI= 10.1145/108844.108850
  19. [DA01] Dey, A. K., Abowd, G. D., and Salber, D. 2001. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum.-Comput. Interact. 16, 2 (Dec. 2001), 97-166.
  20. [DS03] Diaper, D. and Stanton, N. The Handbook of Task Analysis for Human-Computer Interaction, CRC Press, 2003
  21. [Dx89] A. J. Dix (1989). Software engineering implications for formal refinement. Proceedings ESEC'89, Ed. C. G. &. J. McDermid. Springer-Verlag. 243-259. http://www.hcibook.com/alan/papers/ESEC89/
  22. [Dx99] A. Dix (1999). Hazy, Crazy, Lazy Days are Over - time for designers to think. keynote Designing Information for Mobile and Broadband Network Users, London, 15th December 1999.  http://www.hcibook.com/alan/papers/mobile-and-b-99/
  23. [DB00] Dix, A., Beale, R., Wood, A.: Architectures to make Simple Visualisations using Simple Systems. In: Proc. of. Advanced Visual Interfaces (AVI 2000), pp. 51–60, ACM Press (2000)
  24. [DR00] Dix, A., Rodden, T., Davies, N., Trevor, J., Friday, A., and Palfreyman, K. 2000. Exploiting space and location as a design framework for interactive mobile systems. ACM Trans. Comput.-Hum. Interact. 7, 3 (Sep. 2000), 285-321. DOI= 10.1145/355324.355325
  25. [DF04] A. Dix, J. Finlay, G. D. Abowd and R. Beale (2004) Human-Computer Interaction, third edition. Prentice Hall.
  26. [Dx06] A. Dix, The brain and the web: intelligent interactions from the desktop to the world. In Proceedings of VII Brazilian Symposium on Human Factors in Computing Systems, IHC'06, vol. 323. (2006) 142. DOI=10.1145/1298023.1298080
  27. [DC06] Dix, A., Catarci, T., Habegger, B., loannidis, Y., Kamaruddin, A., Katifori, A., Lepouras, G., Poggi, A., and Ramduny-Ellis, D. 2006. Intelligent context-sensitive interactions on desktop and the web. In Proceedings of the international Workshop in Conjunction with AVI 2006 on Context in Advanced interfaces (Venice, Italy, May 23 - 23, 2006). CAI '06. ACM, New York, NY, 23-27. DOI= 10.1145/1145706.1145710
  28. [DK07] A. Dix, A. Katifori, A. Poggi, T. Catarci, Y. Ioannidis, G. Lepouras and M. Mora, From Information to Interaction: in Pursuit of Task-centred Information Management, in Proc. of DELOS Conference 2007, Pisa, Italy, 2007.
  29. [DK08] A. Dix, A. Katifori, G. Lepouras (2008). Globally Accessible Local URIs - discussion ideas. http://www.hcibook.com/alan/projects/TIM/docs/gloi/
  30. [Dx08] Dix, A. 2008. Tasks = Data + Action + Context: Automated Task Assistance through Data-Oriented Analysis. In Proceedings of the 2nd Conference on Human-Centered Software Engineering and 7th international Workshop on Task Models and Diagrams (Pisa, Italy, September 25 - 26, 2008). P. Forbrig and F. Paternò, Eds. Lecture Notes In Computer Science, vol. 5247. Springer-Verlag, Berlin, Heidelberg, 1-13. DOI= 10.1007/978-3-540-85992-5_1
  31. [DK10] Alan Dix, Akrivi Katifori, Giorgos Lepouras, Costas Vassilakis. Spreading Activation Over Ontology-Based Resources: From Personal Context To Web Scale Reasoning. International Journal of Semantic Computing, Special Issue on Web Scale Reasoning: scalable, tolerant and dynamic (in press) 2010.
  32. [Do09] L. Dodds, Understanding the Big BBC Graph, dated 11th June 2009. Accessed 10/08/2009. http://blogs.talis.com/n2/archives/569
  33. [Do08] The Dojo Foundation. The Book of Dojo, 1.3. http://docs.dojocampus.org/
  34. [DM07] M. Dzbor, E. Mottaa, and J. Dominguea, (2007). Magpie: Experiences in supporting Semantic Web browsing. Web Semantics: Science, Services and Agents on the World Wide Web, 5(3), Pages 204-222
  35. [En95] Endsley, M. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64.
  36. [EK95] Ericsson, K., and Kintsch, W. 1995, Long-term working memory. Psychological Review, 102, 211-245.
  37. [FL06] Faaborg, A., Lieberman, H.: A Goal-Oriented Web Browser. In: Proc. of the Conference on Human Factors in Computing Systems (CHI 2006), pp. 751–760, ACM Press (2006)
  38. [FY05] Fallman, D. and Yttergren, B. 2005. Meeting in quiet: choosing suitable notification modalities for mobile phones. In Proceedings of the 2005 Conference on Designing For User Experience (San Francisco, California, November 03 - 05, 2005). Designing For User Experiences, vol. 135. AIGA: American Institute of Graphic Arts, New York, NY, 55.
  39. [HD96] Hall, W., Davis, H., Hutchings, G..: Rethinking Hypermedia: The Microcosm Approach. Kluwer Academic Publishers, Norwell, MA, (1996)
  40. [HS07] M. Hartmann, and D. Schreiber: Prediction Algorithms for User Actions. Workshop on “Lernen, Wissen und Adaptivität" (Learning, Knowledge, and Adaptability), LWA 2007, German Society for Informatics, pp. 349-354
  41. [Ha03] M. Hasan, A Spreading Activation Framework for Ontology-enhanced Adaptive Information Access within Organisations. in Proc. of the Spring Symposium on Agent Mediated Knowledge Management (AMKM 2003), Stanford University, California, USA, 2003.
  42. [Ha09] M. Hausenblas. Having your profile forms automatic filled in … Blog post dated 5th April 2009. http://webofdata.wordpress.com/2009/04/05/pac-intro/
  43. [HD05] Tom Heath, Martin Dzbor, and Enrico Motta (2005) Supporting User Tasks and Context: Challenges for Semantic Web Research. In Proceedings of the Workshop on End-user Aspects of the Semantic Web (UserSWeb), European Semantic Web Conference (ESWC2005), Heraklion, Crete. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS//Vol-137/11_heath_final.pdf
  44. [HM05] Tom Heath, Enrico Motta, and Martin Dzbor (2005) Context as Foundation for a Semantic Desktop. In Proceedings of the 1st Workshop on The Semantic Desktop, International Semantic Web Conference (ISWC2005), Galway, Ireland. http://tomheath.com/papers/heath-motta-dzbor-semdesk2005-context-semantic-desktop.pdf
  45. [HM08] T. Heath and E. Motta (2008) Revyu: Linking reviews and ratings into the Web of Data. Journal of Web Semantics, Vol. 6 (4).
  46. [HH85] Hutchins, E. L., Hollan, J. D., and Norman, D. A. 1985. Direct manipulation interfaces. Hum.-Comput. Interact. 1, 4 (Dec. 1985), 311-338.
  47. [HH00] Hollan, J., E. Hutchins and D. Kirsh (2000) ‘Distributed cognition: toward a new foundation for human-computer interaction research’. ACM transactions on computer-human interaction, 7(2), 174-196.
  48. [HZ08] Hussein, Tim; Ziegler, Jürgen: Adapting web sites by spreading activation in ontologies. In: ReColl '08: Int. Workshop on Recommendation and Collaboration (in conjunction with IUI 2008) Gran Canaria (2008)
  49. [Hu83] Hutchins E. Understanding Micronesian navigation. In D. Gentner & A. Stevens (Eds.), Mental models. Hillsdale, NJ: Lawrence Erlbaum, pp 191-225, 1983.
  50. [Hu95] Hutchins E. Cognition in the Wild. MIT Press, 1995.
  51. [Ia01] Renato Iannella, Representing vCard Objects in RDF/XML, W3C Note, 22 February 2001. http://www.w3.org/TR/vcard-rdf
  52. [JT07] W. Jones, J. Teevan (eds), Personal Information Management, University of Washington Press, 2007.
  53. [KV08] A. Katifori, C. Vassilakis, I. Daradimos, G. Lepouras, Y. Ioannidis, A. Dix, A. Poggi, T. Catarci, Personal Ontology Creation and Visualization for a Personal Interaction Management System, in Proc. of PIM Workshop, CHI 2008, Florence, Italy, 2008.
  54. [KV09] A. Katifori, C. Vassilakis and A. Dix, Ontologies and the Brain: Using Spreading Activation through Ontologies to Support Personal Interaction. Cognitive Systems Research (in press) (2009)
  55. [Ku98] Kushmerick, N. (1998) , (Toward) an extensible wrapper repository standard. Workshop on AI & Information Integration, AAAI-98 (Madison).
  56. [LD06] G. Lepouras, A. Dix, A. Katifori, T. Catarci, B. Habegger, A. Poggi, Y. Ioannidis (2006). OntoPIM: From Personal Information Management to Task Information Management Personal Information Management, SIGIR 2006 workshop, August 10-11, 2006, Seattle, Washington.
  57. [Li01] Lieberman, H.: Your wish is my command: programming by example. Morgan Kaufmann, San Francisco (2001)
  58. [LW05] W. Liu, A. Weichselbraun, A. Scharl, and E. Chang, Semi-Automatic Ontology Extension Using Spreading Activation, Journal of Universal Knowledge Management, 0(1) (2005), 50 – 58.
  59. [MM04] McBride, C. and McKinna, J. 2004. The view from the left. J. Funct. Program. 14, 1 (Jan. 2004), 69-111.
  60. [M02] Gordon Mohr. MAGNET v0.1, (Created 2002-06-12; Revised 2002-06-17, Accessed June 2008). http://magnet-uri.sourceforge.net/magnet-draft-overview.txt
  61. [NM98] Nardi, B., Miller, J., Wright, D.: Collaborative, Programmable Intelligent Agents. Communications of the ACM, 41(3), 96–104 (1998)
  62. [PK97] Pandit M., Kalbag, S.: The selection recognition agent: Instant access to relevant information and operations. In: Proc. of Intelligent User Interfaces (IUI 97), pp. 47–52 ACM Press (1997)
  63. [RV97] P. Resnick and H. R. Varian (Eds.), Special Issue on Recommender Systems, Communications of the ACM, 40(3) (1997) 56–89
  64. [RS04] E. Rukzio, A. Schmidt, H. Hussmann, Privacy-enhanced Intelligent Automatic Form Filling for Context-aware Services on Mobile Devices, in Proc. of workshop on Artificial Intelligence in Mobile Systems 2004 (AIMS 2004), in conjunction with UbiComp 2004, Nottingham, UK, 2004.
  65. [Sa05] L. Sauermann, The Gnowsis Semantic Desktop for Information Integration, in Proc. of the 3rd Conference Professional Knowledge Management, Kaiserslautern, Germany,2005.
  66. [Sa08] Leo Sauermann (ed.). Desktop URIs. (accessed May 2008). http://aperture.wiki.sourceforge.net/SemdeskUris
  67. [Sh82] Shneiderman, B. (1982). The future of interactive systems and the emergence of direct manipulation. Behavior and Information Technology, 1, 237-256.
  68. [SH08] L. Sauermann and D. Heim, Evaluating Long-Term Use of the Gnowsis Semantic Desktop for PIM, in Proc. of the 7th international Conference on the Semantic Web, Karlsruhe, Germany, 2008, 467-482.
  69. [SM04] Stylos, J., Myers, B., Faulring, A.: Citrine: providing intelligent copy-and-paste. In: Proc. of the 17th Symposium on User Interface Software and Technology (UIST 2004), pp. 185–188, ACM Press, (2004)
  70. [Su87] Suchman, L. (1987). Plans and Situated Action, MA: Cambridge University Press.
  71. [Sw03] Luke Swartz (2003). Why People Hate the Paperclip: Labels, Appearance, Behavior, and Social Responses to User Interface Agents. Honors Thesis, Symbolic Systems Program, Stanford University
  72. [WD97] Wood, A., Dey, A., Abowd, G.: Cyberdesk: Automated Integration of Desktop and Network Services. In: Proc. of the Conference on Human Factors in Computing Systems (CHI 97), pp. 552–553, ACM Press (1997)
  73. [WM89] Wright P. C. and A. F. Monk, A cost-effective evaluation method for use by designers. International Journal of Man-Machine Studies, 35, 1989.
  74. [WD96] W3C draft, Client Side Automated Form Entry, W3C Working Draft WD-form-filling-960416, http://www.w3.org/TR/WD-form-filling.html
  75. [WE10a] W3C Editor's Draft. Web Storage. dated 4th Feb. 2010. http://dev.w3.org/html5/webdatabase/
  76. [WE10b] W3C Editor's Draft. Web SQL Database. dated 4th Feb. 2010. http://dev.w3.org/html5/webdatabase/

http://www.hcibook.com/alan/papers/JWS-2010-world-of-action/

Alan Dix 27/4/2010