Shared Annotation for Cooperative Learning

James R. Davis1 and Daniel P. Huttenlocher2

1 Xerox Corporation, Design Research Institute. 2Computer Science Department, Cornell University

Abstract
CoNote is a system that enables a group of people to communicate via shared annotations on a set of electronic documents.  The system is implemented using the World Wide Web, and operates with any standard Web browser supporting forms.  This paper describes the conceptual model for collaboration in CoNote, and discusses our experiences using the system in an introductory college computer science course.  We have found that shared annotations of documents provide a richer communications forum than electronic media such as newsgroups, bulletin boards and email distribution lists.  The central difference is that the documents provide a context for discussions, thereby enabling people to find relevant information more easily.

Keywords — annotation, World-Wide Web.

CoNote is a computer supported cooperative work system intended to facilitate communication within a work group through shared annotations (marginal notes) on a set of documents.  The central idea is that shared annotations provide an effective communications forum for groups whose work involves frequent reference to some set of documents (e.g., teachers and students, field service workers, editors and publishers, standards organizations).  In our experience, the shared annotations model provides a richer electronic forum than media such as news groups, bulletin boards or mailing lists.  The key difference is that the documents being annotated provide a context for group discussions, thus enabling people to find discussions on particular topics more easily.  The shared annotations model also provide a more structured forum than shared authoring tools, because the documents play the role of a fixed context for the discussions rather than being mutable themselves.
We have been experimenting with the CoNote shared annotation system since the fall of 1994, and have used it as the primary delivery vehicle for lecture notes and problem sets in Cornell's Computer Science 212 (CS212).  We have found that students are able to easily locate relevant information because most of the dialog is in the form of questions and answers referring to a particular problem on an assignment.  In particular, using CoNote we have rarely seen the same question being asked by different students, because they are able to find and understand other students' questions and answers.  In this paper we describe the conceptual model for collaboration in CoNote and discuss our experiences using the system for teaching.
Overview of CoNote
CoNote is based on the World Wide Web, and is thus accessible to anyone with a Web browser supporting forms.  No special client software is required, documents are read and annotations are viewed and posted using an ordinary browser such as MacWeb, Mosaic, or Netscape.  Documents and their annotations may be written in plain ASCII text or in HTML.  In contrast to other Web based annotation systems, such as Mosaic, CoNote annotations appear in-line at the place that they refer to rather than at the end of the document. Actually, only links to the annotations appear in the document; the annotations can be accessed by navigating this link structure. Numerous word processing systems permit annotations (e.g., Microsoft Word and Adobe Acrobat) but in such systems the annotations modify the document, and are not shared.  In contrast, CoNote annotations are seen by all members of a work group.  In the CoNote system, documents and users both belong to groups, which provides various levels of access control (as discussed in the following section).
Unlike paper annotations, CoNote annotations can't go just anywhere in a document, but only at places that the author of the document has designated as appropriate.  We refer to each such place as an  annotation point.  When you are reading a document and reach an annotation point, you see links to all the existing annotations at that point.  Each link shows the title (or subject) of the annotation, the name of the person who made it, and the date. The annotations at a given point are arranged into a tree, so that replies to a given annotation appear immediately below and indented with respect to it.  Thus the structure of the display reflects the reply structure of the conversation.
 
Problem 1

There are 7 annotations
Primitive Function returned from call to conference Student 1 Thu, 22 Sep 94 17:07:04 EDT
		BUG IN ps3 SOURCE CODE?????? Student 1 Thu, 22 Sep 94 22:47:46 EDT
		Defining cs-advisor TA Fri, 23 Sep 94 11:32:42 EDT
stategies Student 2 Sun, 25 Sep 94 16:45:39 EDT
		BUG WITH POWERMAC Professor Sun, 25 Sep 94 18:43:03 EDT
My stupidity hasn't improved.  Student 3 Mon, 26 Sep 94 17:42:19 EDT
		Don't use ' Student 4 Mon, 26 Sep 94 18:16:30 EDT
You may add an annotation

Define an advisor named cs-advisor that uses some meaningful combination of the sample strategies we gave you in the code.  Call the conference function with your cs-advisor and the interactive-student.  For Problem 1, turn in the listing of your cs-advisor definition and a brief sample of the advisor working correctly.
Figure 1.  How annotations appear within a document.  The annotations list is delimited by horizontal rules.  Note the tree structure: the second and third annotations are both replies to the first, and are indented accordingly.  

An example of how an annotation point is displayed is shown in Figure 1.   The original document being displayed contains the text above and below the two horizontal rules.  The rules and the text between them are inserted by the CoNote system, because the author has specified an annotation point just after the text "Problem 1''. There are a total of seven annotations at this annotation point, three of which annotate the document itself and the remainder of which are replies to other annotations.  The first annotation has two replies to it, which are so indicated by indenting them one level (replies to replies would be further indented, and so on).  The "smiley face'' icon indicates annotations made by an author of the document; these are referred to as authoritative annotations.
The annotation titles are active links.  Selecting a link displays the annotation in full. An annotation display shows the title, the name of the person posting the annotation, the date and the annotation text.  Following the text are links to those annotations that refer to this one (if any) and those annotations that this one refers to (if any).  This allows the reader to move through a conversation using information about the subject, author and date of each related annotation.  Below the list of related annotations are three triangular navigation buttons, pointing left, upwards, and down.    The left and right triangles take the reader to the chronologically previous and next annotation at that annotation point. Thus it is possible to follow the discussion at a given point in chronological order, rather than in terms of the tree which encodes the reply structure.  The upward triangle returns the reader to the document itself, at the annotation point where the current conversation is rooted.
Users add annotations by filling out a form, specifying a subject and text.  The date and name of the user are filled in automatically by the system.  By default, annotations are entered in plain text, but people knowledgeable in HTML may enter HTML formatting codes if they wish.  Each annotation point has a link to the form for adding an annotation, which is denoted by the link add an annotation. There is also a "Reply'' button on each annotation.  Annotations entered as replies are automatically linked to the annotation that they pertain to.
In addition to accessing annotations when reading or browsing documents, it is often useful to be able to search for annotation information according to attributes such as when the annotation was posted, what document it was posted on, who the authors were, etc. Thus there is a forms-based search interface that searches a given set of documents for annotations matching some query, and then displays a tree structure of just those annotations. The selected annotations can then be accessed in the normal fashion.  
Conceptual model
The CoNote model of the collaboration process is based on the concept of a  document group, which is a set of people who share a collection of documents.  Each person in a group has a certain  role with respect to the documents in that group.  The possible roles are  viewer,  reader,  user and  author, where each of these roles has more access to the documents than the one before.  A viewer may view an annotated document, but can't see the annotations on it.  A reader can also see the annotations, but can't add any. A user may read and add, and an author may read, add, and also delete annotations.  In addition, annotations added by an author are considered authoritative and marked as such .
A person may have a different role in different groups.  For example you might have the role of an author in one group and be a reader in another.  Groups can also define a "default'' role, providing some level of access to unregistered users.  If there is no default role, then unregistered users are not permitted any access to the documents or annotations in the group.
Although annotations appear to the user in-line in the document, they are stored in a separate database, and are interpolated into the document when it is delivered to the user.  Thus the document itself is not changed in any way by the annotations, and indeed it is usually stored read-only.  This also means that the same document can appear in more than one document group, where each group has a disjoint set of annotations on the document.
A document group consists of a set of users, each with a specified role, and a set of documents.  The set of users is specified by a  roles definition, which lists each user and the role he or she has for that document group.  The documents of the collection are each specified by a  document definition, which names the document, lists the location in the file system where the text is stored, and may also list the locations of the annotation points within the document (see below).
  Each document in CoNote has a unique name, the  docid, which is a composite of the names of the document group and the document definition.  Further, each annotation point has a name unique within the document.  The author assigns these names when designating potential sites for annotation.  Finally,  annotations at a point are numbered chronologically. 
Experience
We have used the CoNote system in Computer Science 212 (CS212), since the Fall of 1994.  This class had about 90 students last semester, and has about 65 students this semester.  All of the lecture notes, handouts and problem sets for the class are available on the CoNote Web server.  The definitive versions of the problem sets are the online versions; that is, students are responsible for the information in the annotations, including any corrections or clarifications posted by course staff (document authors).  This tends to focus activity on the online versions of problem sets, particularly as the due dates approach.  For example, in the first 57 days of the current semester (from Jan 20, 1995 to March 17, 1995), 428 annotations have been made, and these have been read 7704 times. The distribution of the number of annotations read over time shows a strong correlation with the due dates of problem sets, as illustrated in Figure 2

Figure 2.  Annotations read per day by problem set due date.  Note the double dip for problem set two, which was postponed.  
The 73 users of the system have read an average of 105 annotations apiece in this two month time period.  Some users read only a few, while a few read nearly all of them. Figure 3 shows the distribution of how many users read a given number of annotations.

Figure 3.  Number of users who read between n and n+30 annotations.   The peak (16) shows that most people read between 30 and 60 annotations, with the number falling off more or less linearly.  Three users read 240 to 270 annotations.
In using CoNote, we have observed a number of interesting things.  One of the authors (Huttenlocher) has taught CS212 for 6 years.  The Fall 1994 semester, in which we used CoNote for the first time, had a considerably tighter grade distribution than in any other semester he has taught the course.  In particular, there were many fewer students at the tail end of the grade distribution.  Of course this could merely be random fluctuation, but there is anecdotal evidence that the shared annotations were particularly helpful to the students at the lower end of the grade curve.  One of the main problems for such students is that they get stuck on the problem sets, which are large programming assignments.  Often they fail to complete these assignments, which are nearly half of the grade in the course.  This appeared to be much less prevalent.  Moreover, several students commented in office hours that they found the annotations useful because it showed them that other students were also confused.  Simply knowing that they were not alone seemed to make a big difference.
The annotations were clearly a part of the course materials, students often referred to annotations when asking questions in lectures, recitations or office hours.  Students also developed high expectations about the turnaround time for having questions answered. They would often send email to course staff pointing out that they had an unanswered annotation from a few hours ago. Prior to the use of CoNote, it would have been unrealistic for them to expect answers in less than a day (as they would have to find a course consultant, TA, or the professor).  It should be noted that many of the questions posted on the problem sets were answered (correctly) by other students.  Thus, while the system did put an extra load on the course staff in terms of having to check for unanswered questions in a timely manner, it did not necessitate staff answering all of the questions.  
Students used CoNote not just in the laboratory, but also from home (dormitories or off-campus).  Table 1 shows the number of accesses to CoNote from various locations.
CS212 lab room	27186	
dormitory net	15328	
Cornell dialup 	7641	
CS department	5624 	
other Cornell 	2520	
unknown or outside 	146	
total accesses	58445 	
The course laboratory room was the most used site (about 46% of the accesses), but nearly 40% of the accesses came either from the dormitory network or the campus dialups (from students living off campus or in dorms not yet wired to the net).  Approximately 25% of the dormitory rooms on campus have ethernet connections (the remainder will be activated this summer).  Thus, we would expect that dorm access will increase further in the fall, as many students currently are not directly connected to the campus net.
A major role of CoNote has thus been to enable students to work as a group from their dorm rooms, rather than having to go to a special laboratory with many machines.  It provided opportunities for "conversation'' outside of the ordinary class- and lab-room locations.  Many students listed this as a major advantage of CoNote on the course evaluation forms from last semester.  Another advantage from the instructors' point of view is that CoNote provides all the students with the same access to information.  When many students are working in the lab room, subgroups tend to share hints and information.  Often these hints are wrong, and students can waste a lot of time as a result.  The fact that the annotations are available to the entire class, and that both students and staff can correct misunderstandings, significantly improves on this.
Conclusions
Thus far, CoNote appears to be quite useful in teaching and learning. We have found that the shared annotations model enables people to find relevant discussions easily, by embedding the discussions in the context of a set of documents.  The system is being actively used by students in CS212, and has become an integral part of the course.  It is particularly encouraging to us that students have often been able to answer each other's questions.
We see a number of opportunities for further investigation.  For instance, we could post each student's solutions after the due date, allowing students to compare their approach with that of others and to critique the style and format of other solutions.  Just as in writing texts, there is much to be learned by reading and commenting on writings by other students.  Second, we have a number of suggestions from the students to consider.  One common request was the ability to delete one's own notes.  We find a number of annotations are later followed by "oops'' messages.  Should these publicly visible mistakes be removed on the grounds that they are clutter, or should they remain, as encouraging evidence that even the best students (and even course staff) are sometimes confused?  Third, we'd like to apply CoNote to other kinds of collaborations.  In the teaching environment, there's a clear distinction between the authors (course staff) and users (students), in that authors are assumed to be authoritative. While this may be so in teaching, it is less likely true in other collaborative contexts.  Fourth, we'd like to consider annotation of other media types besides text, e.g.  graphics.  Finally, we are working on a reimplementation of CoNote which will be both faster and more secure than the current version.
Further information about CoNote can be found on the CoNote home page: http://dri.cornell.edu/pub/davis/annotation.html.

Authors Addresses
Davis: Xerox Corporation, Design Research Institute, Cornell University, Ithaca, NY 14853, davis@dri.cornell.edu
Huttenlocher: Computer Science Department, Cornell University, Ithaca, NY 14853.  dph@cs.cornell.edu