The Espoo intranet AI assistant experiment brought new understanding about information search and the utilisation of generative AI

The experiment involved 20 testers from different functions: communications, project and programme office, HR and financial services. The pilot was coordinated by experts from the Experimental Development team of the Digital Services Development and Knowledge Management unit in cooperation with IT Management. The implementation partner was Espoo Intranet publishing system contractual partner Twoday Oy.

“The experiment provided an excellent opportunity to see in practice what generative artificial intelligence requires in order to succeed – from both data and procedures. We also learned valuable lessons about how solutions should be defined and how significant the impact of the quality of source data is. The AI Assistant is not just a tool, but its utilisation also changes procedures. This is not just about technology,” says Development Manager Valia Wistuba, who led the experiment.

The experiment was carried out in a separate, protected Azure environment where test users were able to test the assistants at guided test events. A limited number (28 pcs) of instructions and documents were included in the tailored GPT assistant, whereas the MS Copilot and intranet search used by Espoo rely on the information content of the actual use environment.

Using the test environment, it was possible to assess differences from the perspectives of functionality, quality and costs. “During the pilot, GPT worked well. Cost monitoring provided valuable information to Espoo and helped understand the different elements of additional costs,” says IT Development Manager Marko Piippo.

• Artificial intelligence makes work more efficient – when the data is high-quality
  According to the users, the GPT assistant speeded up information search, content production and instruction production for daily work when the source data was clear and up-to-date. However, in complicated situations, clarifications and better source references were needed. It was possible to search for things by asking questions, which meant that exact keywords were less necessary.
• Tailored assistant increases confidence in the use of artificial intelligence
  Particularly in tasks requiring clarity and conciseness, GPT was considered the easiest solution. The pilot increased understanding about the reasons for the hallucinations of the AI assistant and how these can be reduced by specification.
• Understanding of the AI assistant cost structure increased
  Cost monitoring showed that operating costs were moderate and that there are opportunities for optimisation.
• Espoo’s intranet search functioned in the detailed search, but it does not support conversational information search
  The search engine provided long lists of links, which may take time to go through. Using the intranet often requires knowledge of its structure and the correct terms. The search does not support the user in creating new content and handles each search separately.

• Source data determines success
  Fragmented instructions, image-based content and varying formats weakened the quality of AI responses. Clear, machine-readable content is a prerequisite for reliable responses.
• Data volume has a direct effect on costs and quality
  The limited amount of data in the pilot helped the cost monitoring, but raised the question of how the solution will scale to the scope of the entire intranet.
• Users need support and learning
  The formulation of the right questions, the assessment of sources and understanding the operating logic of artificial intelligence emerged as important factors.
• Artificial intelligence changes working methods, not just technology
  Conversational search and automatically compacting information change roles, processes and employees’ ways of searching for information.

The results of the experiment will be shared at presentation events of the Espoo organisation as well as in various networks. The lessons learned will support the assessment of future AI solutions, the development of content production and the specification and deployment of AI assistants.

“Although the tested solution will not be taken into production, the experiment increased Espoo’s capabilities to utilise generative AI safely, cost-effectively and in a user-oriented manner,” says Director of Communications Johanna Pajakoski, the owner of the experiment.

---

The City of Espoo’s aim is to promote the digitalisation of municipal services, utilisation of new technology, introduction of electronic services and adoption of new operating methods. These are the goals we are implementing in the Digital Agenda programme.

Further information: Digital Agenda | City of Espoo