Australian Taxation Office (ATO) > Enterprise Content Management System:
Failure to Launch: Learning About Design
the Hard Way

We differentiate between three editorial levels of stories:

1) Design thinking classics:
Case studies that are well-documented and widely known, which we include in our collection for the sake of completeness. If not stated otherwise there are compiled by our editors via desk research.

2) Normal cases:
Stories, which are less known and got collected and rewritten by our editors via desk research.

3) Stories with validated data:
These cases are based on first-hand empirical information that our editors received during their research.

In 2004, the Australian Taxation Office (ATO) decided to design and build an enterprise content management system to streamline its publishing process. The first attempt revealed the technology of the day was inadequate for the design of this complicated project. However, 12 years later the ATO now has a world class website which is shaped by daily user feedback and longitudinal unmoderated usability benchmarking.

T he Australian Taxation Office is a leading public sector design organisation. It has a long and established history of using design thinking and employs a significant design capability. However, in the early days, the marriage between the traditional areas of information technology (IT), business process development and design thinking was a tumultuous, if not a grueling struggle. The discord of competing methods and philosophies created a string of campaigns between rivalry factions within the organisation. Battle weary foot soldiers: designers, programmers and business analysts wrestled over the role of user-centered design and the credibility of using real users to test designs before they were implemented. Applying design thinking to a complex business process, with new systems and capability was a risky but ultimately fortuitous weapon against potentially disappointing and unacceptable public sector outcomes.

Further still, little consideration was given to employees and their use of the newly designed systems. External users, taxpayers or “clients” were involved early in the design process, but staff were thought of as a resource that could be trained and would learn the process, systems or capabilities. Usability and “designing for natural systems” were foreign concepts for products or services that effected employees.


Enterprise content management system – much more than a tool

One of the significant projects for the ATO during the highly publicised, Easier Cheaper More Personalised Change Program – a complete transformation of the ATO’s IT systems, was an innovative enterprise content management (ECM) system. This tool was expected to revolutionise the way documents were authored, managed and published to the public. Each existing document often included content from approximately 10-20 sources within the organisation, depending on technical specialization and knowledge of the tax areas. This tool would enable multiple documents to be created from “chunks” of information from each source and provide concurrent publishing to multiple channels. Massive savings in labour and consumables were estimated from the delivery of this product as well as fewer manual interventions, improved governance, better accuracy, consistency of content and faster publishing times.

The project team were well engaged with the ATO’s administrative design methodology (Integrated Administrative Design), and were also strong advocates for involving users in the process. However, they believe that the technology would shape the business process (as it was leading “best practice” at the time) and the project significantly invested upfront in perfecting the conceptual design. The project created a core design team which included a range of senior decision makers and managers. The presence of these powerful stakeholders meant that organisational obstacles could be overcome quickly and with minimal effort. A robust design blueprint of the new service was created out of a number of extensive workshops.

The project team were ambitious. They wanted to change the role of authors, the process, tools and outputs in a single release. Not impossible – but definitely challenging given the remaining implementation time of three months.

What happened?

Deadlines for implementation were looming and only minimal users of the process or tools (that is the authors or editors) had been involved in the design process. The core design team had made many of the design decisions based on ideal conditions with little input on the practicalities of real world authoring. Critical IT architecture had also been finalised, which meant only superficial changes to the user interface of the tool and minor process modifications could be made.

The project engaged the Simulation Centre, ATO’s custom built usability and collaborative design studio, to test the design of the model, simulating the activities, tools and environment that would approximate the new live ECM system. This was one of the two biggest fully operational simulations the ATO has conducted to date. It required the full development of role statements, scripts and activities for each participant in the process. Each part of the process was role-played by real users with activities or tasks to perform.

Linear process model of the publishing process designed by the core design team

Diagram 1.1 – Linear Interaction Model of the ECMS publishing process as tested in the simulation

Diagram 1.1 depicts the linear interaction model used for the simulation. The model itself is based around a compound document architecture requiring the content owner (Information Co-ordinator) to have a clear understanding of what will be produced before anything is written. Authors receive allocated parts of the master document for completion.  Once complete, they send their content back for review and editing.  Eventually the document is reconstructed for final review and senior officer sign-off.

The simulation itself was undertaken in two iterations only, both over two day periods. The compressed timeframes restricted any further sessions. Many parts of the process were run concurrently in adjoining rooms with different design facilitators including an authoring workshop, interaction of participants with a paper prototype, mock-call centre assistance calls as well as many small two person meetings. As many as 30 participants and facilitators were actively involved at any time in multiple sessions. Mock training was also given to users to ensure their knowledge of the tool would approximate that of the real world.

The output from each two-day simulation was a draft Goods and Services Tax (GST) guide for the racing industry. The product would be created by the simulation team as a low fidelity prototype as imagined output from the publishing tool.

Overall, the final versions of the product: The Racing Guide, were very poor quality. This result was not due to the prototyping process, the simulation team expected crudely assembled prototypes. However, the fragmented authoring approach created immense issues for the users, making the authored content clunky, disjointed and difficult to read. The simulation uncovered problems relating to major cultural and conceptual changes in the authoring process. All users were frustrated and confused. Staff roles were ill-defined and overlapped. There was also a lack of workflow and task support (from both the ECM tool and the business process). Finally, the paper prototype of the ECM tool user interface had very poor usability and did not match the way the users created content.


The process itself looks very clean – but the problem is that it is not a linear process – it’s organic. You start at a point and then you spread out. You need to consider this before this [model creation]. In an ideal world, it could work like this, but generally, a product just comes together in the steps it comes together.

Employee participant in simulation, the role of an “author”


There were also a number of significant business risks identified. Some examples include: reduced productivity; poor quality publications (fragmented, lack of cohesion, inappropriate level of user language or tone, etc); increased workarounds outside of the tool (and hence lack of audit control); increased use of external artefacts by users; potential duplication of work effort; potential increase in duplicate publications; increase in cognitive load for the users (creating possible fatigue); downgrading of some roles leading to poor staff morale; negative response by staff to new procedures which seem “unnatural”; and over a poor client experience.

The simulation also raised questions about the effect of the quality of the publication for the external users: citizens and taxpayers. However, this aspect was not tested during this process.

Where did it go wrong?

The new business process in combination with the implementation of compound document architecture and the publishing tool, were too rigid and restrictive for the creation of quality content. Authoring was seen by many of the participants as a freeform, fluid or organic process. Writing was iterative and could go in any direction at any time. Participants saw the process as creative, so they needed access to the full document context to craft coherent narrative. The ECM tool forced users to artificially generate a document structure before the content had been developed.  The document structure was then used as the basis of the information architecture which would be “componentised” (compound document architecture) by the tool. The tool was inflexible: contributors could not make changes in content structure; and ad-hoc iterations or changes in workflow were unavailable.

The fragmentation of the content into components for authoring was also seen as a conceptual problem for several of the participants. Although some of the participants were able to author their piece of the document without the context of the whole document, some participants found this incredibly difficult, most believed their piece of the document lost its context when placed back into the larger publication. Technical reviewers also expressed concern at the fragmentation of the document for review and authoring. They said that elements reviewed out of context meant something completely different when placed in a broader context. Technical reviewers were anxious that they might make inadvertent mistakes which would produce inaccuracies or misleading information for the public.

The impact of this rigid structure and control imposed upon this workflow would create bottlenecks in process, user workarounds, documents which lack cohesion and lower quality content publications.

Issues Identified

Control of the publishing process was a key goal of the project and this was achieved. However, the features of the model created many new and unanticipated issues, including:

  • unfamiliar roles which were very different to the existing users;
  • a different and inflexible content creation process (“I found it very alien – to existing work practice”- participant);
  • the process did not allow for the necessary iteration required for documents authored by multiple contributors;
  • no user control and freedom;
  • increased technical capability required: all roles needed to be trained in the uses of the tool, as well as a new conceptual authoring model;
  • a fragmented view of content ownership from different parts of the organisation; and
  • cultural issues around attempting to institute too much change at once.

Furthermore, the linear model was not adaptable to the different types of publications nor to the different demands of publishing in a variety of channels. A more constrained and rigid model meant each separate publishing category (i.e. letters, brochures, internet, booklets, call-centre scripting) required a different process.

What was recommended?

The core design team worked with simulation staff to address the issues identified through the testing process. Many iterations of the model, roles and the tool were developed. The final recommended spiral model (diagram 1.2) was based on a human centred design methodology, which acknowledges the iterative and creative aspect of writing.

spiral publishing model

Diagram 1.2 – Proposed Spiral Model

The simulation also revealed that the process could simplified and managed with only a small number of contributors requiring access to the publishing tool. This would make the change process quicker and more manageable.

In addition, the spiral model provided: visibility, accountability and consistency of content ownership; flexibility of process and design style; differentiation of process (or project management) and technical expertise; reduction in the size of cultural change and necessary training. The simulation team made many recommendations about the publishing tool interface. However, the technical constraints meant that many necessary changes were not possible.

What was the outcome?

Unfortunately, due to the timeframes and the magnitude of the change required, the implementation of the proposed ECM system was abandoned. The simulation revealed that the design would not deliver a solution that would be suitable for the users of the system.

The failure of the project team to recognise the importance of the timing and role of users in the design process came at a great cost. Lack of understanding of real world use, the users and their goals, meant critical information about the design was missed.

All of the parts of the design that involved humans were considered to be something that could be added or modified after the “core tool” was built. The core design team had been focused on the technology first, believing in the adage “build it and they will come”. The management at the time thought they could overcome any user unwillingness by mandating the use of the tool. However, they hadn’t accounted for the indirect cost of lost productivity, rework and the subsequent capability investment required to equip the staff to manage a tool that should ironically, simplify their jobs.

The core design team did not include the right mix of people to understand and define the problem. Important senior stakeholders were involved, but they did not have sufficient information to ensure they made favourable, educated decisions. They were a well-intentioned group who failed to think broadly, test their thinking early and be open to the possibility that they didn’t have all of the answers.

Where are we now?

The outcome of the ECM project meant that the organisation persevered with the existing publishing systems and tools for a number of years. The ATO also undertook a number of capability reviews, restructuring and incremental redesign of the various approaches to publishing. Significant investment has been made in improving the content of publications and understanding the needs of end-users (citizens and taxpayers). The ATO has also reconsidered its approach to developing a range of products and interactions and this has been documented and shared with the public in the blueprint for organisational change: Reinventing the ATO. The blueprint acknowledges the important role of participants or users of services and how they are included in the design process.

Importantly, there is also a blueprint of the new experience for staff. The inclusion of employees and understanding their role in providing service to the community is a considerable step forward in the organisational culture and demonstrates maturity of design thinking. The problem has also been redefined, showing the wisdom of lessons learnt, experience and acceptance. The journey continues…

 

  • observation room
    From the observer room

Coming soon: Advocating for users through metrics: usability benchmarking ato.gov.au…

 


If you liked "Australian Taxation Office (ATO) > Enterprise Content Management System", these stories might also be of interest to you:


Image Sources

  • failuretolaunch-diagram1-image1b: ThisIsDesignThinking.net | © Traditional Copyright
  • Spiral publishing model developed from observing participants in the publishing simulation: ThisIsDesignThinking.net | © Traditional Copyright
  • failuretolaunch-slide1-image1: ThisIsDesignThinking.net | CC BY-NC 4.0
  • failuretodeploy-slide3-image3: ThisIsDesignThinking.net | CC BY-NC 4.0
  • failuretodeploy_slide2-image2: ThisIsDesignThinking.net | CC BY-NC 4.0
  • failuretodeploy-header-image1: ThisIsDesignThinking.net | CC BY-NC 4.0

The Authors

Jo'Anne Langham

Please rate this

Trackbacks/Pingbacks

  1. Design Thinking in the Public Sector - Global Project Solutions - […] really needed? According to th…

So, what do you think ...?

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Pin It on Pinterest