AustLII Home | Databases | WorldLII | Search | Feedback

University of Tasmania Law Review

University of Tasmania Law Review (UTLR)
You are here:  AustLII >> Databases >> University of Tasmania Law Review >> 2013 >> [2013] UTasLawRw 4

Database Search | Name Search | Recent Articles | Noteup | LawCite | Author Info | Download | Help

Curran, Liz; Crockett, Andrew --- "Measuring Legal Services: A Practical Methodology for Measuring the Quality and Outcomes of Legal Assistance Services" [2013] UTasLawRw 4; (2013) 32(1) University of Tasmania Law Review 70


Measuring Legal Services: A Practical Methodology for Measuring the Quality and Outcomes of Legal Assistance Services

LIZ CURRAN AND ANDREW CROCKETT[∗]

Abstract

In this article, the authors contend that the hallmark of a positive or ‘successful’ outcome in the context of legal assistance services is not only the case outcome itself, but the quality of the service process preceding the case outcome. Once the characteristic features of a quality service process are identified then the extent to which these characteristics are evident in day-to-day legal practice becomes the most appropriate way of measuring and ensuring positive service outcomes. Outcomes identified in the research need to be within the control of a legal assistance service and can appropriately be used as a measure of service quality and successful outcomes. Indicators of what make a successful or quality outcome in legal assistance services were identified in the research and used to benchmark whether the service was delivering a quality legal service. The article suggests that the approach taken in this research can be utilised and adopted in other areas of human service delivery with modifications to suit the particular service. It is noted that the participatory research approach adopted is not overly burdensome or expensive.

I INTRODUCTION AND CONTEXT

This article discusses research into measuring the quality and outcomes of legal services. The research was undertaken by the author, Liz Curran, for the Legal Aid Commission of the Australia Capital Territory (‘LAACT’) in the latter part of 2011[1] as part of LAACT’s strategic plan for 2008-2012 and in the context of a national review of Commonwealth funded legal assistance services conducted under the National Partnership Agreement on Legal Assistance Services (‘NPA’). [2]

The objectives of the research were to:

1. develop an understanding of the type and complexity of LAACT’s services;

2. define service outcomes and develop appropriate measures of service quality;

3. develop survey instruments to collect information about service quality and service outcomes that enables the value of legal assistance provided by LAACT to be demonstrated and informs continuous service improvement; and

4. pilot the instruments by means of a ‘snapshot’[3] survey of services delivered by LAACT over a two week period.

A National Partnership Agreement on Legal Assistance Services (‘NPA’)

The NPA is a four-year agreement created under the Intergovernmental Agreement on Federal Financial Relations and the Federal Financial Relations Framework. It commenced in July 2010.

The objective of the NPA is a national system of legal assistance[4] that is integrated, efficient, cost effective and focused on providing services for disadvantaged Australians in accordance with access to justice principles of accessibility, appropriateness, equity, efficiency and effectiveness. The NPA specifically relates to the funding and delivery of Commonwealth funded services by state and territory legal aid commissions under the Commonwealth Legal Aid Program. It is directed at legal aid commissions providing efficient and cost effective legal aid services for disadvantaged Australians in accordance with Commonwealth legal aid priorities in order to achieve the following reforms:[5]

a) earlier resolution of legal problems for disadvantaged Australians that, when appropriate, avoids the need for litigation;

b) more appropriate targeting of legal assistance services to people who experience, or are at risk of experiencing, social exclusion;

c) increased collaboration and cooperation between legal assistance providers themselves and with other service providers to ensure clients receive ‘joined up’ service provision to address legal and other problems; and

d) a strategic national response to critical challenges and pressures affecting the legal assistance sector.

B NPA Review

One of the terms of the NPA is that the agreement be reviewed by 30 June 2013. While the NPA sets out the Commonwealth’s funding contribution and reporting requirements for legal aid commissions, it also sets out the broad objectives and outcomes of the legal assistance system which covers the four program areas funded by the Commonwealth: legal aid commissions, community legal services, Aboriginal and Torres Strait Islander legal services and family violence prevention legal services.

The Commonwealth sees the NPA review as an opportunity to review legal assistance as a national system and to develop an evidence base to inform future government decisions relating to the legal assistance sector as well as an opportunity to assess progress of the parties to the NPA towards achieving the following outputs: [6]

1. legal assistance providers increasing the delivery of preventative, early intervention and dispute resolution services;

2. comprehensive legal information services and seamless referral for preventative and early intervention legal assistance services within each state and territory;

3. delivery by state and territory legal aid commissions of efficient and cost effective legal aid services provided in accordance with the NPA and consistent with access to justice principles of accessibility, appropriateness, equity, efficiency and effectiveness, including:

i. preventative legal services such as community legal education, legal information and referral;

ii. early intervention legal services such as advice, minor assistance and advocacy other than advocacy provided under a grant of legal aid; and

iii. dispute resolution services, duty lawyer services, litigation services and post resolution support services.

A review of legal assistance services of this nature and with the stated aims has not been undertaken on such a scale in Australia since the early 1990s. Some reviews have been undertaken since on a state-by-state basis and by the services themselves, but these reviews are of a different nature and scope to the NPA Review.[7] The NPA review encompasses the four program areas funded by the Commonwealth and is being conducted by Allen Consulting Group on behalf of the Commonwealth Attorney-General’s Department.

The scope of the review includes an assessment of the appropriateness of the NPA mechanism for administering Commonwealth legal assistance funding, the adequacy of the existing NPA performance indicators and recommendations for new indicators and benchmarks, and options for outcome and performance indicators for each program area under future funding agreements.[8]

The expected outcomes of the NPA review are to:[9]

a) measure the impact of the reforms identified in the NPA;

b) measure the efficiency and cost-effectiveness of the legal assistance and how integrated the service delivery is;

c) measure the contribution of legal assistance to the COAG reform agenda (workforce productivity and participation and social inclusion objectives);

d) establish an evidence base for policy development, funding decisions and service delivery across the legal assistance sector;

e) establish an evaluation framework for legal assistance going forward;

f) set the framework for negotiations of the next National Partnership Agreement by assessing the appropriateness of current objectives for service delivery;

g) identify cost efficiencies to be achieved through better targeting of resources within the legal assistance sector, which is critical in the tight fiscal environment; and

h) identify the most efficient and cost-effective service delivery models for the legal assistance sector.

It is hoped that the Curran research will inform and help ensure that the outcome and performance indicators adopted by the Commonwealth following the NPA review are realistic, reflect the diversity of legal assistance services and service providers and do not impose an unreasonable reporting burden on the already stretched legal assistance sector.

C Diversity of Legal Assistance Services

The commencement of the NPA in July 2010 signalled a shift in the operations of legal assistance services from a more traditional legal service model towards a more holistic service model. It will take time for the NPA’s objectives of social inclusion, joined up service, client centred and holistic approaches to problem solving to be embedded in practice.

Legal assistance services provide a diverse range of services[10] to people on low incomes and other vulnerable and disadvantaged people and employ a range of service approaches depending on the needs of their particular target client groups in urban, regional and remote settings.[11] Accurate measurement of service quality and outcomes must reflect this diversity and take account of the different timeframes within which service outcomes occur.

There is concern within the legal assistance sector that outcome and performance indicators developed in the NPA review may fail to adequately reflect this diversity and may increase the already onerous reporting requirements imposed on legal assistance services without adding value. Governments should not add to accountability burdens by imposing new layers of reporting unless the data collected provides information that can be used by service providers to inform service delivery while creating an accurate picture for funders and the public. This is especially the case given that legal assistance services work in a long acknowledged context of under-resourcing and underfunding.[12]

D Measuring Service Effectiveness

While the Curran research was timely having regard to the NPA review it was also pertinent because of the trend internationally for governments to expect human services agencies of all types to demonstrate ‘effectiveness,’ ‘impact’ and ‘successful outcomes’. The approach adopted in the Curran research to measuring services is capable of being replicated by other legal and non-legal service providers and adapted to their particular services, circumstances and client groups. Importantly the research approach tries to capture the complexity of service delivery while being easy to undertake in-house at minimal cost.

E Defining the Outcomes to be Measured

There are difficulties in trying to measure outcomes or effectiveness for lawyers if these are based on case results. Lawyers may not always be successful in seeking a desired outcome for a client for reasons unconnected with the skill and effort of the lawyer and beyond the lawyer’s control (this is discussed in more detail later in this article). For example, the evidence against a client may be strong yet the client may not be satisfied when they lose the case. For this reason, outcomes in the context of legal services must be defined according to factors that are within the lawyer’s power to control. The Curran research suggests that the solution is for feedback sought from clients to focus on the characteristics of the service rather than on whether the client considers that they won or lost the case.

The Curran research demonstrates that there are methodologies that can define and measure outcomes and effectiveness but only if care is taken in defining the outcomes to be measured prior to undertaking the task. The research reveals the complex and technical nature of legal assistance work and that even within one legal assistance service specific programmatic responses are required to be delivered differently due to different legislative and policy settings. For example, the criminal law system operates within an adversarial system, whereas in the family law context, policy and legislative settings are about minimising conflict, the best interests of the child and safety.

Government policy and funding decisions which are based on poorly defined outcomes or poorly framed instruments of measurement or performance targets and indicators which are not relevant to the outcomes they purportedly measure, can compromise the effectiveness of a service or program. For example, if a lawyer secures housing for a client in a case, it does not mean it can be extrapolated that a successful outcome of the services’ delivery is to secure housing for clients. The outcome should be defined according to the role or function of the lawyer, which may be to advocate and tell their client’s side of the story, negotiate or to hold authority to account. To suggest that a successful legal service outcome is the securing of housing with the provision of housing subject to so many extra-legal factors (such as available housing stock and waiting lists) over which a lawyer can have little or no control would set the legal service up to fail. Great care is therefore needed to ensure that the outcomes governments want to measure are realistic and within the actual function and control of the legal aid service.

F Characteristics of Quality Service

Based on the research findings the authors contend that the hallmark of a positive or ‘successful’ outcome in the context of legal assistance services is not only, or even predominantly, the case outcome itself, but the quality of the service process preceding the case outcome. Once the characteristic features of a quality service process are identified (for further detail refer to the research report)[13] then the extent to which these characteristics are evident in day-to-day legal practice becomes the most appropriate way of measuring and ensuring positive service outcomes. Outcomes identified in the research that are within the control of a legal assistance service and can appropriately be used as a measure of service quality and successful outcomes are set out in Table 1.

Another cautionary note is that in the drive for efficiency, service effectiveness can be compromised or impeded[14] by insufficient understanding of why services are provided the way they are, and what works well and why. When services are being provided in a challenging and complicated contextual setting (as is the case with many legal assistance services) then quantitative data, on its own, can be misleading. For example, in Victoria statistics revealed that reports of family violence had increased substantially in recent years.[15] On the face of it this might lead to a conclusion that there is more family violence. However a deeper analysis and qualitative examination of the statistics reveals that a proactive policy and legislative environment in Victoria to keep women and children safe from violence, and improved training and intervention by the police, support and legal services and the courts, is leading to further reporting and the granting of protection orders.

G Disadvantage and Social Exclusion

A key focus of legal assistance services in Australia is on addressing social exclusion and targeting services to the disadvantaged. ‘Disadvantage’ in the context of legal assistance services is described in the Curran research as involving or including the presence of one or more of the following factors:

• drug addiction

• mental illness

• language difficulty

• illiteracy

• intellectual disability

• Indigenous (aboriginal or Torres Strait Islander)

• refugee/asylum-seeker

• recent immigrants

• people from low socio-economic backgrounds

• inter-generational disadvantage including people with a poor education, low income, poor health, no-one with a job in immediate family and a lack of connectedness

• chronic disease

• ill health

• fragmented or non-existent family support

• age – either very young and hence vulnerable or elderly and frail

• gender – for example issues of domestic violence and impacts often statistically affect more women than men

• sexuality – Often people who are transgender, gay or lesbian can experience compounding difficulties including discrimination on the basis of their sexuality.

• HIV/AIDS/Hepatitis

• institutionalisation from a young age

• repeated prison stints in adult- youth life cycle

• unwanted/unloved

• conflict of cultures

• domestic violence

• child abuse victim

• physical disability

In many cases the clients of legal assistance services may have more than one of these issues.

The complex nature of legal assistance service clients is recognised in international research.[16] This considerable body of research on access to justice, legal need and advice seeking behaviours reveals that the work of legal assistance services is subject to significant variability depending on the client group and systemic intervening factors which a service is unable to influence. Mowles, Stacey and Griffin warn that there are often ‘unanticipated, contextual and contingent unforeseen circumstances’ and ‘unexpected ambiguity, paradox and the power differentials and imbalances at play’.[17]

The Legal Australia-Wide (‘LAW’) Survey,[18] a national survey of legal need undertaken by the Law and Justice Foundation of NSW, found that people with a disability have a significantly higher prevalence of legal problems and often experience multiple and substantial legal problems. Other vulnerable and disadvantaged groups in the community with a high prevalence of legal problems identified in the LAW Survey include unemployed people, single parents, people in housing for the disadvantaged, people whose main income is government payments and Aboriginal and Torres Strait Islander people.

In looking at disadvantage and social exclusion it is important to note that some clients:

• cannot read or write;

• cannot speak the language or read and write in their own language;

• may not be amenable to responding to written requests for information such as people with a mental illness;

• live in remote and isolated parts of the country, are homeless or have multiples of these issues.

A large proportion of legal assistance clients fall into categories of disadvantage and so the use of surveys and written questionnaires may mean these clients miss out on giving feedback altogether. Given they are the principal target of legal assistance services valuable insights from these clients’ perspectives would be missed if selected methodologies for feedback or evaluation did not include their input. Focus groups, interviews or forums appropriate for the relevant target client groups to give feedback might be appropriate in these cases.

II THE LITERATURE

The literature consistently states that to be effective measures/indictors need to be:[19]

• relevant

• useful and measurable

• achievable

• practical to measure

• within the service or practitioner’s control and influence.

The United Nations Development Program (‘UNDP’)[20] has developed a series of indicators that are a useful starting point for measuring legal assistance service outcomes, suggesting the following:

1 the defining of capacity development strategies;

2 defining baselines for each indicator; and

3 defining targets for each indicator.

The UNDP’s ‘User’s Guide’ provides a table on ‘Inclusion, Participation, Equity and Empowerment’ enabling environments[21] which could be utilised in any examination of legal assistance services’ efforts to avert social exclusion, community development and in the conduct of community legal education evaluations in Australia.

In making the research relevant and realistic the World Bank Handbook recommends a participatory approach involving key stakeholders.[22] Why? Because setting goals in isolation from what is being done and what has to be done and by those who do it might lead to a ‘lack of ownership’ on the part of the main internal and external stakeholders. The World Bank Handbook recommends a participatory and consultative process be taken to all stages of the identification of goals, objectives, what outcomes look like and the steps necessary to get there i.e. building the indicator system. This approach is integrated into the Curran research methodology discussed below.

The trend towards ‘outcomes’ measurement has been met with some trepidation in Europe.[23] This research notes there can be a tension between ‘top down’ and ‘bottom up’ approaches. Efforts to balance the need for a national picture but draw meaningful conclusions from the measurements can come into conflict. Care is needed in extrapolations from the aggregating of data as distortions can result.

The World Bank Handbook warns against approaches that try to set indicators in a vacuum from what the desired outcomes might be because it is the outcome not the indicators that will produce the best benefits. This is why it is not advisable to use measures such as time costing or activity reporting which are commonly used in commercial legal practices unless those approaches are exact measures for demonstrating what the organisation is seeking to achieve and whether it has achieved it. They may measure time and the number of actions taken, but do they demonstrate that such time and activity was of a kind that led to a quality outcome that aligns with the strategic aims and objectives of the organisation? [24]

Ebrahim and Rangan state that measures of outcomes/results need to align with the actual goals of the organisation and be reasonably within the control of the organisation to influence. They note that ‘organizational efforts beyond this scope are a misallocation of scarce resources’.[25] They warn that a key challenge in such measurement lies in the ‘thorny issue of causality: impacts are likely to be affected by multiple factors and multiple actors.’[26] Ebrahim and Rangan state that ability to attribute long term results to interventions is severely limited as so many other factors can be involved.

Ebrahim and Rangan also discuss the need to gather data that is meaningful and has purpose; otherwise it becomes a time and resource intensive but useless exercise.[27] They also warn that any attempt to measure outcomes must be aligned with an agency’s strategy and mission and the systems and measurements which support such alignment. They warn that profit driven corporations can be easier to measure as what is often examined and compared is savings and profits rather than impacts upon human lives.[28] Social outcomes can be difficult to gauge especially with limited resources and the limits to the ability to measure outcomes. Given legal assistance services in the Australian context deliver human services this warning is highly relevant as people’s lives are affected by the nature of the legal advice given. The international literature warns about the significant difficulties involved in measuring access to justice.[29] Ebrahim and Rangan also warn that many outcomes can often only occur over a longer time than most accountability measures are concerned with. This is consistent with the findings of Smith and Patel which will be discussed later in this section.[30] These warnings from the literature were heeded in the Curran research and were carefully crafted into definitions of outcomes discussed below.

One of the performance standards for Community Legal Advice Centres in the United Kingdom is that the outcomes of cases achieve ‘substantive benefit’ for the client. Whether a client has received a substantive benefit is based on the outcome reported for the client.[31] This new development in the United Kingdom has seen the Legal Services Research Centre (‘LSRC’)[32] tackle issues around measuring performance and outcomes in recent times. They are cautious about such measurement and have also underlined the need for caution, realism and care.

Smith and Patel of the former LSRC stated that any evaluation report must be read in conjunction with the other client focussed studies, which together provide a more comprehensive picture of what services have delivered for clients.[33]

A Cautions around Client Survey Feedback

Client ‘satisfaction’ surveys are often used for market research into services such as clothing or cosmetic retailing or help-lines for computer customers. Often they relate to a service received online or by phone by a telecommunications company or financial service.[34] Borrowing approaches from such industries (unless they are very limited in scope and there is simplicity of service transaction) are likely to distort findings. Use of research nomenclature is important because using yardsticks such as ‘satisfaction’ can distort people’s perception of what is being evaluated. In the legal services context the role and function of a lawyer is confused if ‘satisfaction’ is used as the yardstick since the service will not always ‘satisfy’ clients because legal advisers cannot be a mere ‘mouthpiece’. This is explicitly discouraged by the professional conduct rules, and duties owed to the law, the courts and the administration of justice take precedence over duties to the client. Professional rules of conduct have developed over time to protect clients as well as to balance the interests of individuals with the public interest in the proper administration of justice. There is a danger when the evaluation tools for measurement of legal services require a ‘satisfied’ or ‘not satisfied’ response as the very the process of asking these questions can further mislead the client participating in the survey in their understanding of the role and function of their lawyer. This is because the questions imply that lawyers are there to ‘satisfy’ the clients rather than give independent and fearless advice: advice that may not always ‘satisfy’ the client.

In using evaluation models developed in other contexts care is needed in the design of questions to ensure their relevance to legal assistance services. Many of the studies examined that used the phraseology of ‘client satisfaction’ were from private law firms whose focus was on commercial marketing. It is concerning that some legal aid instrumentalities have also utilised this nomenclature.[35]

The international ‘client satisfaction’ studies analysed in the Curran research highlight the danger of approaches to seeking feedback from clients which fail to take sufficient account of the function and duties of lawyers. Some clients will not like what they hear from a lawyer who is providing competent and independent advice. The client may believe that they are right and the other side is wrong, that the law should not work the way it does, or that the lawyer’s role is to be their mouthpiece. It is not uncommon for such a client to shop around for legal advice that suits them and then when no lawyer will do so, often find themselves self-represented.[36] The fact that lawyers as officers of the court have duties relating to the integrity of the law and the legal system that are higher duties than those owed to the client is not commonly understood in the wider community.[37] This lack of appreciation of a lawyer’s role can be a cause of client dissatisfaction and is why using client satisfaction as a yardstick for measuring quality or effectiveness is problematic in relation to legal services.

Careful processes to measure client feedback are needed and the methodology developed in the Curran research seeks to extract information in a way that acknowledges and overcomes some of the problems and tensions outlined above.

III THE RESEARCH

Informed by the literature discussed above, the Curran research was undertaken between August and December 2011 and the research findings published in late April 2012.[38] The evaluation model not only incorporated learning from previous Australian and international research into measuring service outcomes but, most importantly, was informed by discussion of the nature of legal assistance services - their layers, complexities, contradictions and impediments - with the people delivering the services.[39]

A participatory action research methodology was used. This means that the research alternated between action and critical reflection on what lessons can be learned and how the approach may be modified to incorporate the lessons learned along the way. By doing this the later cycles continuously refine methods, data and interpretation in the light of the understanding developed in the earlier cycles.[40] Under the guidance of the researcher, LAACT staff became designers and stakeholders in the research by making suggestions and critiquing the methodology. This ensured that the research was informed by those who do the work and see day-to-day the complexities and challenges of the legal system and the place of client within it. The research also incorporated goals and strategies from LAACT’s strategic plan. The project used focus group discussions to identify the service objectives of each of LAACT’s practice areas before examining outcomes and the indicators of what was required to achieve those outcomes.[41]

There has been little previous research that undertakes the actual measurement of the outcomes or quality of publically funded legal assistance services in the Australian context and so the research commenced as a trial of survey instruments and to establish a baseline against which to measure future progress.[42] Data for the trial was gathered by means of a ‘snapshot’ survey of family and criminal law services delivered by LAACT between 9 November and 23 November 2011. The idea behind the snapshot approach is to reduce the burden of continuous data collection and reporting on service providers and generate comparable and useful data over time by periodically conducting surveys of a representative sample of services over a fixed period.

The research was conducted in two phases. Phase One involved:

• the collation and analysis of relevant national and international research , LAACT’s strategic documents, including its strategic and operational plans;

• an introductory conversation with LAACT staff;

• focus group discussions with legal, paralegal and support staff in LAACT’s Legal Practice, Knowledge Services and Client Services areas to examine what each service area does, its role and function, some of the ‘outcomes’ and what is required to achieve these, and the nature of the client groups and stakeholders;

• feedback from a former LAACT client; and

• the development of instruments to measure service quality and outcomes informed by the above processes.

The multiple methodologies (including Phase Two below) were also designed to enable research tools to gain responses that could be checked and verified against each other and enable different stages and parts of LAACT’s activities to be examined and measured against quality and outcome indicators. This involved engaging during the research period with service providers, clients and external stakeholders to obtain their respective viewpoints on LAACT’s services and to assess relationships and interfaces with clients, courts, other parties, significant networks and community agencies. In this way the research presented a ‘360 degree’ view of services from those involved in and affected by the service’s operations.

Phase Two was a two-week trial ‘snapshot’ survey of services in the family and criminal practice areas between 9 November and 23 November 2011 using the following instruments and methodologies developed in Phase One:

• eight lawyer and eight client interviews were conducted by the researcher (interviews with lawyers and clients were conducted separately following each lawyer/client interview);

• entries in observation logs by seven staff who were not involved in other survey tasks;

• a voluntary client feedback questionnaire handed to all clients after receiving legal advice at LAACT’s office;

• a telephone survey of clients following closure of their case files;[43]

• an online survey (using SurveyMonkey) of in-house lawyers and private lawyers who handle legally assisted cases;

• case studies collected from open questions in observation logs, focus groups, client interviews with the researcher and the online survey; and

• interviews with stakeholders identified by each practice area and with academics from the Australian National University College of Law, who work with law students participating in LAACT’s Youth Law Program and Legal Aid Clinic advice service.

A key concern throughout the project was that the research and future application of the survey instruments should not impose an onerous additional record keeping or data entry burden on staff. The researcher therefore developed a process that can be adapted across a variety of organisations and applied with minimal resourcing. It is designed to be a sustainable method of capturing data on service quality and outcomes by means of periodic ‘snapshot’ surveys that avoids the administrative and cost burden of continuous measurement regimes.

A How the Research was Conducted and What it Revealed

As a result of the Phase One research, 11 legal assistance service outcomes and indicators of what they might reveal about the service were identified. These are shown in Table 1.

Table 1:

Outcome
Qualities demonstrated by Outcome
1. A good client interview.
Holistic, Joined-up, Quality, Problem Identification, Empowerment, Good Practice, Early Intervention, Prevention, Responsiveness, Client Centred, Alternative Dispute Resolution (ADR), Targeting, Expertise.
2. Clients with chaotic lifestyles attend interviews, appointments and court dates.
Early Intervention, Prevention, Empowerment, Client Centred, Holistic, Targeting.
3. As appropriate, sentences are minimised or unsubstantiated charges are dropped.
Rule of Law, Efficiency, Good Practice, Expertise.
4. Clients are better able to plan and organise their legal affairs.
Early Intervention, Prevention, Empowerment, Quality, Good Practice, Client Centred.
5. Improvement in the client’s interaction with the legal system.
Early Intervention, Prevention, Empowerment, Client Centred.
6. Consideration of issues before a court or tribunal enhanced because the lawyer asked questions/raised issues and brought the client’s story before the court.
Rule of Law, Quality, Voice, Flexibility, Good Practice, Client-Centred, Responsiveness, ADR, Expertise.
7. Client is better able to understand their legal position and the options open to them.
Early Intervention, Prevention, Empowerment, Good Practice, Quality.
8. A process is undergone where the client is listened to, respected and given fearless advice of their legal position.
Quality, Client Centred.
9. Relationships and trust building with other legal and non-legal support agencies enabling client referral and support.
Early Intervention, Prevention, Holistic, Joined-up, Good Practice, Quality.
10. Holding of authority to account.
Rule of Law, Quality, Voice, Flexibility, Good Practice, Client-Centred, Responsiveness.
11. A holistic service delivered to the client through collaboration, networking, community legal education and joined-up services.
Good Practice, Client Centred, Problem Identification, Collaboration, Prevention, Early Intervention, Holistic, Joined-up.

Twenty-three case studies that were considered to be examples of successful legal interventions were extracted from the online practitioner surveys, focus groups, client surveys and practitioner logs. The case studies were analysed according to which of the 11 outcomes in Table 1 they represented and the service qualities demonstrated by those outcomes were identified.

The following are a sample of questions used in the Phase Two trial snapshot survey instruments and the responses obtained. For more detailed information on the survey outcomes refer to the full report.[44]

1 Client Interviews

As a result of seeing the lawyer today are you better able to plan and organise your affairs?

Yes (100 per cent)

No (0 per cent)

Do you feel:

a. You understand what to do next? (87.5 per cent)

b. You understand what steps you need to take? (100 per cent)

c. You understand what steps the lawyer will/will not take and why? (100 per cent)

d. You understand all the options open to you? (100 per cent)

Explain:

Sometimes I have to be told over and over again. It must be annoying for the lawyer and their helpers. When I am stressed I can only take some things in. I appreciate the way they tell me in small bites. I can get overwhelmed if it’s too big picture. They give me it in bits so I go step by step. It’s how they know I cope.

He explained really clearly everythings [sic].

2 External Stakeholder Interviews

One comment reflecting many others was:

Legal Aid’s in house lawyers are good to deal with. They bend over backwards to do what they can for their clients. There is often a significant imbalance in what they can do with their resources.

3 Log Books

The log book was kept like a diary and was provided to staff members in the participating practice areas in both booklet and electronic form.

The purpose of the observation log was to identify:

• the nature of interactions that occur in the delivery of legal aid services;

• the complexity and detail of the work that was taking place; and

• how much time the work took.

An example of the personal logs reveals the nature, complexity and time intensive requirements of dealing with clients and navigating the legal system:

Family law solicitor spoke to me about a duty client who needed assistance to have her [baby] daughter returned from a non-Hague convention country. I spoke to the client about support services available to her as a Muslim woman and parent and made a warm referral to a local Imam from Islamic Centre. I had recently met with the Imam to discuss outreach work at the Islamic Centre and speaking with Muslim families about FDR[45] and Australian family law and to invite the Imam to address lawyers and other professionals working in the family law sector about Muslim families. This is an ongoing outreach project with FDR.

Mother obtained legal advice and referral to family support service appropriate to her culture and religion. Within four days the family law solicitor had taken instructions, initiated proceedings and obtained orders that resulted in the mother being able to travel to the Middle East and retain care of her child and return to Australia.

This was an example of FDR and Family law and Client Services (who assisted client with legal aid application form) working together to achieve an outcome for the client. The co-operation resulted in mother and child being reunited in circumstances where the child may have been lost to the mother for many years.

4 Case Studies

Case studies gathered during the research highlighted how dire some of the outcomes for clients could have been ‘but for’ the intervention of LAACT. Here is one example.

One client, with many issues, legal and non-legal, comes to the service sometimes on a weekly basis. She is a very young Aboriginal mother, suffering from alcohol abuse, mental health/co-morbidity issues, and has had little education. She is reliant on Centrelink support and has anger management issues. She loses her temper easily, storms out of office, yells at everyone including those trying to assist, and is unable to receive advice or process information when under stress. While eligible for housing transfer, because of her poor education and personality issues, she has been unable to secure an accommodation transfer with Housing ACT. The lawyer assisted her and wrote a letter to the manager of the relevant area a few weeks ago. Soon after the lawyer’s intervention the lawyer was notified that a caseworker had finally been allocated. The lawyer also referred the client to the family law area for assistance and representation with family law issues. The lawyer is still the main point of contact for her and a lot of the assistance is ensuring referral to appropriate services in a number of areas. In terms of her specific legal problem, the lawyer spent quite some time ensuring the client was clearer on the process, what to expect and her legal rights. Ultimately the legal team was able to negotiate an outcome by consent. The lawyer also liaised with DVCS [domestic violence service] to assist the client to be transported to the court on the relevant day as the client had no money for the bus. Without the advice, the lawyer notes that the client would probably still be stuck in the system litigating in court or having her matter dismissed for non-attendance, would not know what to expect and may be still searching for suitable accommodation for herself and her children.

This case study demonstrates the following service outcomes and qualities:

Table 2:

Outcome
Qualities demonstrated by outcome
Clients with chaotic lifestyles attend interviews, appointments and court dates.
Early Intervention, Prevention, Empowerment, Client Centred, Holistic, Targeting.
A process is undergone where the client is listened to, respected and given fearless advice of their legal position.
Quality, Client Centred.
Relationships and trust building with other legal and non-legal support agencies enabling client referral and support.
Early Intervention, Prevention, Holistic, Joined-up, Good Practice, Quality.
Holding of authority to account.
Rule of Law, Quality, Voice, Flexibility, Good Practice, Client-Centred, Responsiveness.

This case study also highlights the effort, significant support and advocacy interventions that are necessary when clients have multiple compounding legal and social problems. The lawyer must be persistent and, where issues of trust are at stake, consistent. The example also highlights that clients often face an intransigent and non-responsive system that is difficult to navigate and that having an articulate advocate in such cases can lead to improved outcomes.

The LAACT services surveyed in the trial snapshot as part of the research rated highly and consistently on the outcome indicators across the different measurement tools used. This suggests that the 11 desired outcomes and quality service were present. This was verified by clients and stakeholders.

B Further Snapshot Surveys

Since the November 2011 survey LAACT has conducted two further snapshot surveys of different service areas and the extent to which the outcomes defined in the Curran research are being achieved. Like the original survey the snapshots were taken over two-week periods using the survey instruments developed in the research. The service areas surveyed on these occasions received high ratings from clients and other stakeholders. Importantly, however, the surveys have also revealed some aspects of service delivery where there is scope for improvement which has enabled remedial action to be taken.

The LAACT surveys have confirmed research findings in Australia and elsewhere that disadvantaged and vulnerable people often experience multiple legal problems with severe consequences and that unresolved legal issues frequently impact on people’s health and wellbeing.[46] For example, 75 per cent of clients in one survey had other legal problems linked to their presenting problem and identified their legal problems as causing, being linked to, or increasing their stress levels.[47] The surveys have also been a rich source of case studies which illustrate the complexity of legal assistance work and the difference that timely provision of legal advice and other assistance can make to the resolution of legal problems with positive flow on effects to people’s health and wellbeing.

The value of the surveys lies not only in the positive feedback that affirms what the service is doing well, and in doing so sustain staff commitment and morale, but also in any negative feedback that reveals service weaknesses which might otherwise be overlooked and not addressed.

IV CONCLUSION

Mowles, Stacey and Griffin[48] give a salutary warning to funders and agencies concerning the measurement of outcomes. They note that managerial methods have been adopted largely uncritically from the private sector and are now ubiquitous across a range of organisations and in the expectations of funding bodies. They observe that when applied to processes of social interaction like human development or services, such methods have serious shortcomings. These methods overlook or ‘fail to understand unanticipated contextual and contingent circumstances unforeseen in the more abstract and de-contextualised planning processes to be such “noise” which needs to be managed away’.[49] Mowles, Stacey and Griffin also warn of the dangers of applying ‘historical and abstract tendencies’ since the tendencies over-emphasise causality and linearity, implying that human action is more rational and pre-reflected than it actually is.[50] These warnings are highly relevant in the context of the current NPA review of legal assistance services in Australia.

As illustrated by the case study selected from the research and outlined above, the situations in which clients and service providers find themselves are often inherently paradoxical, difficult, unpredictable and affected by systemic issues. Yet despite these challenges quality service delivery can transform the experiences and lives of clients who struggle to navigate a complex legal system and deal with the power imbalances that exist in their relationships with others.

Having worked for many years in community legal centres both authors note that similar complexities characterise the work of other legal assistance services. Legal assistance organisations have limited resources and it is critical that they remain focussed on service delivery. While funders and other stakeholders have a legitimate interest in ensuring that legal assistance service providers are accountable for the efficient and effective expenditure of public funds, accountability measures must take account of the realities of service delivery and not impose an unreasonable burden on service agencies. Snowden and Boone[51] observe that complicated circumstances, unlike simple ones, may contain multiple right and wrong answers. They observe that the more complex and protracted the situation and behaviours are, the more expertise, flexibility and adaptability is required. This relationship needs to be acknowledged and guide decision making by those who fund and manage legal assistance and other human services.

For these reasons, the authors conclude that any attempts to review or evaluate legal assistance services must recognise a need for an understanding of the need for care, the complexity and diversity of service delivery.


[∗] Dr Liz Curran is a Senior Lecturer in the Legal Workshop at the ANU’s College of Law. Andrew Crockett is Chief Executive Officer of LAACT. He was previously a Senior Lecturer in the Faculty of Law at Monash University and a former Director of the Legal Aid Commission of Victoria. Andrew Crockett has an LLB from the University of Melbourne and a LLM from Monash University and a Graduate Certificate in Higher Education from Monash University. Dr Liz Curran has an LLB/BA from Monash University, a Graduate Diploma in Education (Sec) Australian Catholic University, an RSA Certificate from Godmer House, Oxford, a LLM from University of Melbourne and a Doctorate of Juridical Science from La Trobe University.

[1] See discussions about the difficulties of measuring human service outcomes in United Nations Development Program, Overview of the UNDP’s Approach to Measuring Capacity (June 2010) 10 <http://europeandcis.undp.org/news/show/FF9BCD4F-F203-1EE9-B8E6C30BA3F534A0> . The United Nations Development Program highlights the difficulties of the task and the possibility that measuring outcomes/results will not always be an exact science. They stress that measurement must go beyond an increase in input resources and completion of activities. The challenges of outcome measurement in a legal context are discussed in Marisol Smith and Ash Patel, ‘Using Monitoring Data: Examining Community Legal Advice Centres Delivery’ (Report, Legal Services Commission UK, June 2010) <http://www.justice.gov.uk/publications/research-and-analysis/lsrc/research-projects/research-on-community-legal-advice-networks-and-centres/publications>

and Adeline Trude and Julie Gibbs, ‘Review of Quality Issues in Legal Advice: Measuring and Costing Quality in Asylum Work’(Report, Information Centre about Asylum and Refugees and the Immigration Advisory Service, City University of London, March 2010) 1. For a more detailed discussion of the literature see a report undertaken for the Attorney General’s Department by the author, Liz Curran; ‘A Literature Review: Examining the Literature on How to Measure the ‘Successful Outcomes’: Quality, Effectiveness and Efficiency of Legal Assistance Services.’ (Literature Review, Attorney General’s Department, 2012) <http://www.ag.gov.au/LegalSystem/Legalaidprograms/Documents/Literature%20review%20of%20legal%20assistance%20by%20Dr%20Liz%20Curran.pdf> .

[2] Council of Australian Governments (‘COAG’), National Partnership Agreement on Legal Assistance Services (1 July 2010) ( <http://www.ag.gov.au/LegalSystem/Legalaidprograms/Documents/National%20Partnership%20Agreement%20on%20Legal%20Assistance%20Services.pdf> . Information on the Review can be found at: Attorney-General’s Department, Government of Australia Review of Legal Assistance Programs (November 2012) <http://www.ag.gov.au/LegalSystem/Legalaidprograms/Pages/Reviewoflegalassistanceprograms.aspx> .

[3] ‘Snapshot’ research involves conducting the research for a short time but on a regular basis. In this case the snapshot is over a two week period once or twice per year. By undertaking ‘snapshots’, comparisons between periods can still be undertaken.

[4] The broad objectives of legal assistance described in the NPA are to enable Australians to access justice, to assist in closing the gap between Indigenous and nonIndigenous Australians, to assist those at risk of social exclusion, to assist in alleviating homelessness and contributing to the broader COAG reform agenda. Legal assistance also contributes to the earlier resolution of legal problems through increasing the delivery of preventative, early intervention and dispute resolution services and assists in the efficient running of the court system, promotes alternative dispute resolution and facilitates the law reform work undertaken by legal assistance providers.

[5] COAG, above n 2, 4.

[6] COAG, above n 2, 4-5.

[7] For a detailed and comprehensive discussion of other evaluations and reviews see Curran, above n 1.

[8]For the full terms of reference of the NPA review see: <http://www.ag.gov.au/LegalSystem/Legalaidprograms/Pages/Reviewoflegalassistanceprograms.aspx> .

[9] See Allen Consulting Group, Review of the National Partnership Agreement Evaluation Framework (2012) <http://www.ag.gov.au/LegalSystem/Legalaidprograms/Documents/Final%20Evaluation%20Framework.pdf> .

[10] The range of legal assistance services provided by the four program areas include legal information, referral, legal advice, minor legal assistance and advocacy, duty lawyer services at courts and tribunals, grants of assistance for legal representation in criminal family and civil litigation, dispute resolution services, community legal education, counselling, non-legal support in legal proceedings and law reform. The range of services delivered by each provider type varies widely and this diversity presents particular challenges when it comes to developing realistic and accurate measures and comparing like with like.

[11] Community Law Australia, Unaffordable and Out of Reach: The Problem of Access to the Australian Legal System (July 2012) <http://www.communitylawaustralia.org.au/wp-content/uploads/2012/07/CLA_Report_Final.pdf> .

This report notes that 80% of clients of CLCs have an income of $26,000 or less.

[12] Senate Legal and Constitutional Affairs Committee, Parliament of Australia, Inquiry into Access to Justice (2004), ch 8;

Australian Law Reform Commission, Review of the Federal Civil Justice System Discussion Paper 62 (1999); Australian Law Reform Commission, Managing Justice: A Review of the Federal Civil Justice System, Report No 89 (2000).

Beth Gaze and Rosemary Hunter, ‘Access to Justice for Discrimination Complainants: Courts and Legal Representation’ [2009] UNSWLawJl 37; (2009) 32 (3) University of New South Wales Law Journal 699-724; Michael Kirby, ‘Law and Justice in Australia: Room for Improvement’[2004] QUTLawJJl 1; , (2004) 4 Queensland University of Technology Law and Justice Journal 1-13; Mary-Anne Noone, ‘The State of Australian Legal Aid’ (2001) 37 Federal Law Review 29; Ronald Sackville, ‘Access to Justice: Towards an Integrated Approach’ (2001) 10 Judicial Review 221; Louis Schetzer, ‘Community Legal Centres: Resilience and Diversity in the Face of a Changing Policy Environment’ (2003) 31 Alternative Law Journal 159.

[13] Liz Curran, ‘We Can See There’s a Light at the End of the Tunnel Now’: Demonstrating and Ensuring Quality Service to Clients’ (Report, Legal Aid ACT, 2012) 1-2.

[14] Trude and Gibbs, above n 1, 1.

[15] In Victoria, family violence incidents increased 109% between 1999-2000 and 2011. There had been an increase in the laying of criminal charges from 1 in 6 in 1999-2000, to 1 in 3 in 2010-11. Intervention Orders finalised since 1999–2000 increased by 144%.

[16] Alexy Buck, Nigel Balmer and Pascoe Pleasance,‘Social Exclusion and Civil Law: Experience of Civil Justice Problems Among Vulnerable Groups’ (2005) 3 Journal of Social Policy and Administration 302, 318-320; Richard Moorhead, Mark Sefton and Gillian Douglas, ‘The Advice Needs of Lone-parents’ (2004) 34 Family Law 667; Alexy Buck et al, ‘Lone-parents and Civil Law: An Experience of Problems and Advice-seeking Behaviour’ (2004) 38 Journal of Social Policy and Administration 221, 253-269; Sarah Ellison et al, ‘The Legal Needs of Older People’ (Report, Law and Justice Foundation of New South Wales, 2004) <http://www.lawfoundation.net.au/report/older>

Christine Coumarelos, Zhigang Wei and Albert Z Zhou, ‘Justice Made to Measure: New South Wales Legal Needs Survey in Disadvantaged Areas’ (Report, Law and Justice Foundation of New South Wales, 2006) <http://www.lawfoundation.net au/report/survey2006>

Maria Karras, Emily McCarron, Abigail Gray and Sam Ardasinski, ‘On the Edge of Justice: the Legal Needs of People with a Mental Illness in New South Wales’ (Report, Law and Justice Foundation of 2006) <http://xml.lawfoundation.net.au/ljf/site/articleIDs/CB05FD97AAF2458CCA25718E00014293/$file/EdgeOfJustice.pdf> Suzie Forell, Emily McCarron and Louis Schetzer, ‘No Home, No Justice? The Legal Needs of Homeless People in New South Wales’ (Report, Law and Justice Foundation of New South Wales, 2003) <http://www.lawfoundation.net.au/ljf/site/articleIDs/46AFCFE76A8314F8CA25707500815B9F/$file/no_home_no_justice.pdf> Louis Schetzer and Judith Henderson, ‘Access to Justice and Legal Needs, Stage 1, Public Consultations’ (Report, New South Wales Law and Justice Foundation, 2003) < <http://www.lawfoundation.net.au/ljf/site/articleIDs/EA0F86973A9B9F35CA257060007D4EA2/$file/public_consultations_report.pdf> and Law and Justice Foundation of New South Wales, ‘Qualitative Legal needs Survey: Bega Valley (Pilot)’ (Report, Law and Justice Foundation of New South Wales, 2003) <http://www.lawfoundation.net.au/ljf/site/articleIDs/5C64EF48733789C6CA257060007D4F40/$file/bega_report.pdf.> .

[17] Chris Mowles, Ralph Stacey and Douglas Griffin, ‘What Contribution can Insights from the Complexity Sciences Make to the Theory and Practice of Development Management’ (2008) 20 Journal of International Development 804, 809 - 820.

[18] Christine Coumarelos et al, ‘Legal Australia-Wide Survey: Legal Need in Australia’ (Report, Law and Justice Foundation of New South Wales, August 2012) <http://www.lawfoundation.net.au/ljf/app/6DDF12F188975AC9CA257A910006089D.html> .

[19] Dawn Smart, Ask the Expert (2004) USA National Resource Centre < www.ccfbest.org/outcomemeasurements/issuesandchallenges.htm>.

[20] Bureau for Development Policy, ‘Capacity Assessment Methodology User’s Guide’ (Guide, United Nations Development Programme, May 2007) 1 <http://europeandcis.undp.org/uploads/public/File/Capacity_Development_Regional_Traning/UNDP_Capacity_Assessment_Users_Guide_MAY_2007.pdf> .

[21] Bureau for Development Policy, above n 20.

[22] Jody Zall Kusek and Ray C Rist, A Handbook for Development Practitioners: 10 Steps to a Results Based Monitoring and Evaluation System (World Bank, 2004) 58.

[23] Adult Care Information Network (‘ACIN’), Key Messages from the Debate so Far, (2006) The National Archives <http://webarchive.nationalarchives.gov.uk/20080814090357/ic.nhs.uk/our-services/improving-social-care-information/adult-care-information-network> .

[24] Kusek and Rist, above n 22, 58.

[25]Alnoor Ebrahim and V. Kasturi Rangan, ‘The Limits of Non Profit Impact: A Contingency Framework for Measuring Social Performance’ (Working Paper No 10-099, Harvard Business School, May 2010) 4 <http://www.hbs.edu/faculty/Publication%20Files/10-099.pdf> .

[26] Ibid 8.

[27] Ibid.

[28] Ibid 35.

[29] Maurits Barendrecht, Jose Mulder and Ivo Giesen, ‘How to Measure the Price and Quality of Access to Justice’ (Research Paper, Study Group Access to Justice, November 2006) <http://www.tilburguniversity.nl/faculties/law/research/tisco/research/projects/access/papers/06-11.pdf> . See also in relation to civil justice: Jin Ho Verdonschot et al, ‘Measuring Access to Justice: The Quality of Outcomes’ (Working Paper No 007, TISCO Working Paper Series on Civil Law and Conflict Resolution Systems, November 2010) <http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1298917 .> . The latter authors note the ‘problems associated with neutral evaluations of outcomes, the ambiguity of outcomes, and the relative weight of each criterion in different settings.’ Again the Verdonschot et al report’s remit is broader than legal assistance services but the issues they grapple with many of which remain terribly complicated demonstrate the challenges of measuring outcomes and access to justice.

[30] Smith and Patel, above n 1.

[31] Ibid 14.

[32] The LSRC closed on 1 April 2013.

[33] Smith and Patel, above n 1,14.

[34] For example, many ‘free’ trial online ‘client satisfaction’ surveys are made available by various companies such as <https:// about.com>; or <https://www.surveyshare.com/templates/basicdemographics.html>. These may not be appropriately converted to a legal aid service context without considerable care being taken in design and questions for the reasons reflecting the make-up of disadvantage of many legal aid clients referred to earlier in this literature review.

[35] Quality Service Office, Legal Aid Ottawa, LAO Common Measurement Tool Overview of 2009 Results (2010) <http://www.legalaid.on.ca/en/about/downloads/2009_cmt-overview.pdf> 1; Legal Aid Queensland, ‘Annual Report 2010-2011’ (Report, Queensland Legal Aid, 2010) <http://www.legalaid.qld.gov.au/publications/Reports/annual-report/annual-report-archives/annual-report-2010-11/Documents/LAQ-annual-report-2010-11-part-2.pdf> Mary Bacica and Julie Winram, LSS Client Services Survey (3 October 2007) <http://www.legalaid.bc.ca/assets/aboutUs/reports/legalAid/lssClientSurveyResults.pdf> Anna Munday and April Rutkay, Client Satisfaction Survey and Measurement (June 2004) Legal Aid Western Australia <http://www.legalaid.wa.gov.au/LegalAidServices/Documents/Client%20Satisfaction%20Survey%202004.pdf> and Idaho Legal Aid Services, Fair Lending/Fair Housing Client Satisfaction Survey, (2011) <http://www.idaholegalaid.org/article/2088> .

[36] Ainslie Lamb and John Littrich, Lawyers in Australia (Federation Press, 2nd ed, 2011) 155.

[37] See, eg, Legal Profession Act 2006 (ACT); Legal Profession Act 2004 (Vic) and Law Council of Australia, The Australian Solicitors Conduct Rules (at June 2011).

[38] The report including the research methodology, survey data and findings is on LAACT’s website <http://www.legalaidact.org.au/> .

[39] For a discussion of some of the literature on research internationally and some of the problems presented by outcome and quality of legal services measurement see Curran, ‘A Literature Review’, above n 1.

[40] Rory O’Brien, An Overview of the Methodological Approach of Action Research (1998) <http://www.web.net/~robrien/papers/arfinal.html> Gail McCutcheon and Burga Jung, ‘Alternative Perspectives on Action Research’ (1990) 29 (3) Theory into Practice 144-51; Bob Dick, Action Research Tools (May 2011) Action Research, Action Learning <http://www.aral.com.au/resources/arphome.html.> .

[41] Kusek and Rist, above n 22, 59.

[42] Further six-monthly snapshot surveys were conducted in June 2012 and December 2012 using the methodology developed in the Curran research. Thereafter snapshot surveys of all service areas of LAACT are conducted at least annually.

[43] This was the only unsuccessful instrument: client call back rates were low. Staff attributed this to clients’ desire to put their cases behind them and ‘move on’.

[44] See <http://www.legalaidact.org.au/pdf/Legal_Aid_ACT_Quality_Legal_services.pdf.> .

[45] LAACT’s family dispute resolution program.

[46] Coumarelos et al, ‘Legal Australia-Wide Survey’, above n 18.

[47] In the June 2012 survey clients were asked whether the problem they spoke to us about had caused them and/or their family any stress and, if so, how the stress affected their family. They were also asked whether as a result of the advice they received the stress had been reduced, stayed the same, or increased. Responses included: Yes, massive, increased blood pressure and my wife has a bad heart; Feel a bit eased now; Huge amounts, adverse effects to mental, physical and emotional well-being; Stayed the same but feeling more empowered – know my options.

[48] Mowles, Stacey and Griffin, above n 17, 808-820.

[49] Ibid 808.

[50] Ibid 814.

[51] David J Snowden and Mary E Boone, ‘A Leader’s Framework for Decision Making’, (2007) 85 (11) Harvard Business Review, 85. See pp 87 and 89 for a good discussion of understanding complexity.


AustLII: Copyright Policy | Disclaimers | Privacy Policy | Feedback
URL: http://www.austlii.edu.au/au/journals/UTasLawRw/2013/4.html