Experimental Results

The concepts above have been implemented for a commercial electronic mail handler (Eudora) for the Apple Macintosh. The agent, implemented in Macintosh Common Lisp, communicates with the mail application using the AppleEvent protocol. The MBR Engine is domain independent, and can be easily adapted to calendar applications or news readers. Furthermore, all of these applications can share fields and actions. As more applications implement an AppleEvent interface, the agent should be able to aid the user with these applications as well. Currently, several users are actively making use of the agent on their actual mail. While the computations are intensive, we have achieved satisfactory performance on most high end Macintoshes.

The performance of MBR in interface agents has been documented in [Kozierok &Maes1993]. We wish to show that multi-agent collaboration strictly improves upon results obtained from single agent systems. Namely, multi-agent collaboration should steepen the learning curve and improve the handling of entirely novel messages.

To illustrate this, we set up the following scenario using the actual e-mail of two graduate students over a three day period (approximately 100 messages per user).

Calvin and Hobbes are two graduate students in the Intelligent Agents group.

1. Hobbes : Hobbes has been around for some time and hence his agent is quite experienced. Hobbes' agent has noted the following trends in its user's behavior. All messages to `bpm', a music mailing list are refiled to a folder called bpm for later reading. Messages directly addressed to Hobbes are read by him and then deleted, as are messages to other mailing lists.

2. Calvin : Calvin is a new graduate student in the group. Calvin's agent starts out with absolutely no experience. Calvin also refiles all messages from the `bpm' list for later perusal. He deletes subscription requests sent to the list. Calvin reads messages directly addressed to him, and then refiles them to appropriate folders. The rest of his mail, such as messages to other lists, he reads and deletes.

=180pt

We plotted the confidence of Calvin's agent's suggestions as Calvin takes actions on about 100 actual mail messages. Figure 1 shows the results obtained. The x-axis indicates the growing experience of Calvin's agent as Calvin takes successive actions on the mail messages and the number of situation-action pairs in the memory increases. The thick rising trend line indicates how Calvin's agent's performance (in terms of confidence in predictions) rises slowly with experience. The numerous pockets show new user behavior being modeled. The agent makes several mistakes very early, which is to be expected, since the situations it has in its memory early on do not effectively capture all of Calvin's behavior patterns. Towards the end, we see several more mistakes, which reflect a new pattern occuring. With the tell-me threshold for all actions set at 0.1, the graph shows that it will take approximately 40 examples for the agent to gain enough confidence to consistently have suggestions for the user.

=180pt

Figure 2 shows the level of confidence of Calvin's agent in its suggestions with multi-agent collaboration. It may be noted that the confidence levels of all correct suggestions are always greater than the confidence levels generated by Calvin's agent alone at any point. The thick horizontal trend line indicates that multi-agent collaboration enables an inexperienced agent to make accurate predictions with high confidence as soon as it is activated as well as fill in gaps in even an experienced agent's knowledge. Note that trust modelling of Hobbes' agent is taking place inside Calvin's agent with each action Calvin takes on his mail. Space restrictions preclude the presentation of results regarding trust modelling of multiple peers in this paper.