In addition to the numerical data collected above, a questionnaire was also
distributed to the users to get feedback on the subjective aspects of the system.
A copy of the questionnaire is shown in Table 
.
Some of the questions were directly pertinent to the system and dealt with ease
of use and user-friendliness. The other questions dealt with the larger concept
of Interface Agents for Personalized filtering and what the users felt about
the broader issues involved. The answers to the questionnaire are summarized
below. The users were candid about the fact that their responses were mostly
influenced by their background and original expectation levels. This did produce
varied responses from differing perspectives.
People responded quite positively to the graphical user interface. The interface was easy to use for the basic tasks of reading news and providing feedback. The use of colors to indicate association of articles to agents was well received. An area where more needs to be done is in explaining why an article has been selected by the agent. An interesting suggestion is to even provide some of the articles which have been rejected with reasons.
The development of a good ``agent model'' by the users is as important as the agents developing a sophisticated user model. A prerequisite to trusting the agent is understanding its behavior. People had mixed reactions when asked if they could develop good agent models. The visual representation of anthropomorphic agents was quite enjoyable, and easy to understand. However, since the internal state of the agent is quite complex, better tools appear to be necessary to explain the agent's behavior. Users who had previous knowledge of the underlying text processing module found this system to be a powerful tool and could make good use of it. On the other hand, users with no such background had problems correlating the agent's behavior to its internal state and found some agent actions inexplicable.
Two weeks of daily interaction was not a sufficient enough time for some users
to get used to the system, understand the mechanism, train the agent(s) and
see the effects of a well-trained agent. Users who had previous knowledge extracted
a lot of use in a very short time (see figures ,
).
Some users found the system started to get useful towards the end of the testing
phase (see figures ,
),
while others could not judge or perceive any improvement.
In communicating with the agent, one of the problems was the delayed response of the agent. This problem exists because any changes made to the profile causes the agent modify its behavior only after a day, since the new data arrives on a daily basis. This lack of immediate response prevents efficient communication.
There were mixed responses to the question of trusting the agent. Users who understood the agent actions well and knew their abilities and limitations were very comfortable with the idea of trusting their agents for some of their information filtering needs. On the other hand, users who could not understand agent behavior had reservations trusting the agent.
As regards privacy concerns, for the most part users would like to keep the profiles private and only release information that they deliberately wish to. The information maintained in the user profiles can find a number of malicious uses, not the least of which is junk mail. The profiles should probably be safely encrypted and stored locally, to minimize the chances of reaching undesirable destinations.
To summarize the lessons learned from testing in a real environment, this system received an encouraging response from users and the approach appears to be pointed in the right direction. At the implementation level, some more work needs to be done before a system like this finds widespread use. In particular, good explanations have to be provided and the agent internal state needs to be better visualized.