Evaluation of the PRIO Mental Ill Health Action Plan

This report presents a first evaluation of the PRIO mental ill health action plan. PRIO is the government’s strategic drive to combat mental ill health during the period 2012–2016.

BACKGROUND 

There have been several drives in Sweden in the area of mental ill health during the last two decades, yet major challenges remain. Reports have also indicated that mental ill health is on the increase, particularly in young people. Against this background, in May 2012 the Swedish Government passed a resolution on the PRIO mental ill health action plan. A total of SEK 4.3 billion was set aside for the initiative in the years 2012–2016. A leading part of PRIO is an agreement between the Government and the Swedish Association of Local Authorities and Regions (SALAR). The agreement includes stimulus funds founded on a performance-based model and the funds to conduct a number of development projects.

The illustration below summarises the structure and direction of the action plan:

 

 

Overall targets

 

 

  • Equal, knowledge-based, secure and accessible healthcare and care of a high quality
  • Access to work and supported employment
  • Opportunities to be involved and have influence

 

 

 

 

Prioritised target groups and areas of effort

 

 

  • Children and young people
  • People with extensive or complex mental health issues
  • Knowledge support, skills development and quality work
  • Approach, involvement and influence

 

 

 

 

Interim targets

 

 

  • A total of 15 interim targets linked to the prioritised target groups and areas of effort
  • An example of an interim target is to increase patient and user influence in healthcare and care

 

 

 

 

Efforts

 

 

  • Around 40 efforts during 2012
  • An example of an effort is the support to National Cooperation for Mental Health (NSPH) to increase patient and user influence

In order to support development work within the framework of PRIO at local, regional and national level, national coordination functions have been set up at the Swedish Ministry of Health and Social Affairs (Socialdepartmentet) and SALAR. The main task of the coordination function at the Ministry is to coordinate the Government’s efforts, to continuously monitor the results of the investment and – on the basis of this – to take measures to ensure that improvements will be achieved in the areas covered by the action plan. The coordination function at the Ministry will report to a reference and strategy group consisting of the parliamentary undersecretaries of the Ministry of Health and Social Affairs and the Ministry for Children and the Elderly, as well as representatives of the Ministry of Health and Social Affairs, SALAR, the Swedish National Board of Health and Welfare (Socialstyrelsen), and the organisation for patients, users and relatives known as NSPH. The main task of the coordination function at SALAR is to support the implementation of the efforts being made by municipalities and county councils, while also encouraging an exchange of experiences among municipalities and county councils.

VÅRDANALYS’ GOVERNMENT COMMISSION 

Vårdanalys’ overall task is to monitor and assess whether PRIO is producing the intended effects for people with mental ill health. This means monitoring and evaluating the action plan at both action and target level. Vårdanalys should also note any other observations and circumstances that suitably reflect the effects of the investment. Within the framework of the 2013 progress report, which is also the first one, Vårdanalys will report on the following:

  • A proposed plan for the continuation of the monitoring and assessment work in 2013–2017
  • A follow-up of the efforts involved in the investment
  • An assessment of how the work on the action plan and the efforts involved in this have been organised and started

METHOD 

In its work on the current report, Vårdanalys has interviewed representatives from patient and user organisations, the Swedish Ministry of Health and Social Affairs, other authorities, municipalities and county councils, as well as researchers. We have also carried out literature and document studies. Our proposal for an evaluation model has been validated by the expert panel set up by the authority for the validation of PRIO. The expert panel has also submitted its views on the report as it stands. An examination of the facts in this report has been conducted by the coordination functions within the Ministry of Health and Social Affairs and SALAR, as well as by representatives of NSPH and the Swedish National Board of Health and Welfare.

PROPOSED PLAN FOR CONTINUED ASSESSMENT 

Vårdanalys proposes that the assessment of PRIO should be done in six stages as shown in the model below. The evaluation model shows our main issues and the times at which the evaluation will take place.

We will use a combination of quantitative and qualitative methods to respond to the evaluation queries. In general, it can be said that the principal foundation for the analysis in stages 1 and 2 will consist of interviews and literature and document studies. Stage 3 will, for the most part, be based on an analysis of the internal reporting on the efforts involved via the coordination functions of the Ministry of Health and Social Affairs and SALAR. Stages 4 and 5 will initially be based on an analysis of the indicators and measurements drawn up by the coordination function of the Ministry of Health and Social Affairs in order to operationalise the interim and overall targets. More in-depth studies will be carried out in stage 6. These may relate to both individual efforts and/or specific issues, and areas that we consider appropriate as reflectors of the effects of the investment.

Vårdanalys proposes that SEK 4 million per year should be set aside for the assessment during 2014–2016, and that SEK 2 million should be set aside in 2017. The results of the assessments will be published in annual progress reports by no later than 1 October 2014–2016 and in an overall final report by no later than 15 May 2017. The results of the in-depth studies may be published in separate interim reports, or as part of the annual progress report.

2013 ASSESSMENT: RESULTS AND CONCLUSIONS 

Below is a summary of the main results and conclusions from the first assessment, which includes a follow-up of the efforts made in 2012 and an assessment of how the work on the action plan has been organised and started. We will also provide an initial assessment of the extent to which the targets of the investment as they were drawn up are relevant. Vårdanalys will not begin measuring the meeting of targets until the progress report in 2014. This is because the assessment is dependent on the coordination function at the Ministry of Health and Social Affairs establishing the indicators and measurements that will be used to monitor and manage the investment. Such work was started in 2012–2013, and Vårdanalys has submitted its views in the discussion about this.

It should be pointed out that the first year of the investment has been characterised by special circumstances: PRIO and the agreement between SALAR and the Ministry of Health and Social Affairs were not decided upon until May, while the coordination functions were added in August. In other words, the work of 2012 did not cover a full calendar year and the coordination functions had limited opportunities to influence how it was carried out. These limitations have been noted.

  • The targets of the investment and the prioritised target groups and areas of effort address relevant needs and deficiencies 

Vårdanalys considers in general that the choice of overall targets, interim targets and prioritised target groups and areas of effort encompass the relevant needs and deficiencies. However, the wording of some of the targets could be developed and made clearer. For example, the very broad interim target, “To increase knowledge of – and skills relating to – people with extensive or complex mental health issues and their consequences, and the importance of treating somatic illness in people within the target group”, could be defined more clearly and possibly divided into several interim targets. Vårdanalys also suggests that the shortage of housing should be addressed at target level. In addition to this, it should be shown how the various targets relate to one another, and to which targets the chosen efforts are expected to contribute.

  • The continued work to draw up measurable targets and target levels is a pressing issue 

No measurable targets have been formulated in the action plan. The overall targets are of a visionary nature – for example, “Equal, knowledge-based, secure and accessible healthcare and care of a high quality” – while the interim targets may be likened to directional targets, such as “To develop support for relatives of people with mental ill health” or “To create the conditions that will allow the results of healthcare and care to be monitored in an open and comparable way.” Without timed and measurable targets, it is difficult to decide the extent to which the investment has been a success or not. As mentioned above, the coordination function at the Ministry of Health and Social Affairs has begun work on operationalising the targets in the action plan so that it is possible to monitor developments – which is a positive thing. According to the coordination function, the documentation for this work – “Plan for targeted efforts in the area of mental ill health” – will initially form the basis for the management of the investment. In other words, there will be no formal resolution on the document (on the other hand, it will be available and will be communicated externally). Vårdanalys finds the lack of a resolution to be a problem, as the establishing of indicators and target levels would clarify the responsibility for, and direction of, the investment. This is also a requirement if it is to be possible to evaluate the investment.

  • In certain areas, it is considered that the efforts are not sufficient 

Even though all the efforts can be justified individually, it is not clear how the selection process has been arranged and to which of PRIO’s targets the various efforts are expected to contribute. In some areas, there have so far been few efforts, or none at all. For example, there have been no efforts on the theme of somatic illness – despite the fact that one of the interim targets according to the action plan is to “increase knowledge of – and skills relating to – people with extensive or complex mental health issues and their consequences, and the importance of treating somatic illness in people within the target group”. Furthermore, Vårdanalys considers that a majority of the efforts in 2012 initially relate to psychiatric care; fewer efforts have been directed at municipal care and the overall life situation of the target groups. There is no support for this prioritisation in the action plan; here, the emphasis is repeatedly on the importance of the life situation, and especially employment and work. Most of the user representatives whom Vårdanalys has interviewed also express the view that a larger proportion of the efforts should be directed at improving the day-to-day life and overall life situation of the target group. This applies, for example, to efforts to improve the housing situation and access to work and employment.

  • There are deficiencies in the target descriptions of the efforts and in financial reporting 

The accounts of PRIO’s various efforts form an important foundation both for evaluating the results and for deciding how the funds have been used. After examining the information available (primarily accounts of, and decisions on, the efforts), Vårdanalys finds that the requirements placed on the accounts vary and are unclear, and that there is no specified template indicating what they should look like. This has probably contributed to the fact that the accounts of the efforts in 2012 contain deficiencies in the target descriptions and unclear financial accounts. The format and scope of the reports vary greatly. The content and information provided are of varying quality. In several cases, the primary focus is on upcoming work in 2013 rather than what was achieved in 2012. Only a minority of players have established clear and observable targets for the efforts, or described how they have achieved them. In the case of a number of efforts, the targets are simply worded thus: “...the effort must have completed x and y”. Some efforts describe such long-term targets that it is doubtful to what extent it will be possible to evaluate the effects within the set timescale. It is also common for the target descriptions to focus far too uniformly on the various process and structure measurements, without describing in more detail how the achieved targets can improve the situation for the target group. As the basis for the assessments by Vårdanalys primarily consists of the responsible players’ own accounts, deficiencies of this kind limit our opportunity to provide true monitoring.

  • The efforts report varying results 

In view of the limiting factors shown above (the special circumstances that characterised the first year of the investment, the lack of clarity in the links between efforts and interim targets, and deficiencies in the basis for making assessments), it has been difficult to produce an exhaustive evaluation of the extent to which the various efforts have been successful. However, on the basis of the documentation available, the majority of efforts appear to have run as planned, and most of the apportioned funds have been spent. The documentation also shows that the efforts for which final reports have been produced have achieved their targets to a varying extent. The report highlights a number of examples (some better, some worse), of which a number are subject to more detailed comment. Among other things, deficiencies are noted in some of the efforts that form part of the agreement with SALAR, in respect of meeting the targets of the efforts, project planning and reporting.

  • The model of performance-based reimbursement needs to be assessed 

The largest individual efforts within PRIO consist of performance-based reimbursement to municipalities and county councils. Just under 40 per cent (SEK 350 million) of the funds for 2012 have been linked to four targets for performance-based reimbursement within the framework of the agreement with SALAR. The leading theory is that the reimbursement model creates strong incentives to achieve the targets, which can contribute to an overall force for change within municipalities and county councils. This assumption is supported by the representatives of the municipalities and county councils that Vårdanalys has contacted. A majority considers the model to be a good thing to aim for, describing it as effective in the sense that the requirements have led to increased activity in the areas in question. However, it is too early to determine what effects can be achieved in the long term, and the extent to which this may permanently improve the situation for the target group. During autumn 2013, Vårdanalys will report on an analysis of the opportunities and challenges presented by performance-based reimbursement as a Government instrument within healthcare and care. It is considered that there is a great need to continue with assessments in the area.

  • There have been deficiencies in the implementation of performance-based efforts during 2012 

Even though it is too early to make a general statement on the suitability of using performance-based reimbursement as a management tool within the framework of PRIO, we can see that the implementation of the performance-based efforts in 2012 has been deficient in a number of regards.

The basic requirements and various targets in the 2012 agreement on performance-based efforts were worked out jointly between SALAR and a working group within the Ministry of Health and Social Affairs (this should not be confused with the coordination function). It has become apparent in interviews that SALAR and the Ministry consciously chose to opt for relatively low performance criteria. The idea was that the performance targets as initially drawn up would help ensure that the “issues were brought to the table” and that the “work got started”, and that the requirements would be gradually made more stringent in the following years’ agreements. It is still too early to determine how successful this strategy is. On the other hand, Vårdanalys finds there are grounds to question whether the 2012 criteria for the allocation of performance-based reimbursement correlate to a satisfactory degree with the descriptions of the targets. Performance target 2 may be mentioned as an example. According to this, the municipalities and county councils should take part in a preliminary study into how to measure accessibility to first line healthcare for children and young people with mental illness. According to the target description in the agreement, municipalities and county councils would “actively take part in development work, try out creative functions and submit their views on the models produced.” In order to gain approval in accordance with the assessment criteria, it was sufficient for the municipalities and county councils to have reported the name of a reference person and replied to a question on a questionnaire as to whether or not they had a method for measuring waiting times for first line care. In this context, it should be mentioned that the criteria for the 2013 performance-based efforts have been strengthened.

It can also be stated that many municipalities and county councils did not succeed in meeting the 2012 performance criteria, even though these have generally been described as relatively low. For example, only nine county councils met the criteria for performance target 3 (Registration on National Quality Registers), while just over half of the municipalities met performance target 4 (Inventory of efforts directed at people with mental health functional impairment). A shortage of time was probably a contributory factor in preventing more from achieving the targets. The information to municipalities and county councils was sent out relatively late in the year, and a majority of the representatives of the municipalities and county councils whom Vårdanalys asked state that the administration of the performances has been demanding in terms of time and resources. In several cases, it has been necessary to appoint new staff in order to be able to process the reporting within the given timescale. Vårdanalys also finds that the decision-making process for the allocation of performance funds was made more complicated by the fact that the coordination functions (of which the one at SALAR was responsible for informing the municipalities and county councils of the terms and conditions) and the National Board of Health and Welfare (which was to consider whether or not the target had been reached) did not agree on the assessment criteria.

  • The coordination functions occupy important roles in the investment 

The overall assessment by Vårdanalys is that the coordination functions within the Ministry of Health and Social Affairs and SALAR have central roles to play in the investment. Given the complexity and scope of PRIO, it is considered necessary for these players to have a long-term responsibility for managing and developing the investment.

Vårdanalys considers that the work of the coordination functions is being carried out in accordance with their respective assignments. Some positive initiatives that Vårdanalys would like to emphasise are the communication efforts being made by the coordination functions regarding PRIO, and the work being done by the coordination functions at the Ministry of Health and Social Affairs in operationalising the targets of the investment. There are also a number of areas for improvement. One example is the effort reporting for which the coordination function within the Ministry of Health and Social Affairs is responsible.  Vårdanalys has identified deficiencies in the wording of the targets of the efforts as well as in the reporting of results and financial reporting. It is a matter of urgency for the coordination function within the Ministry of Health and Social Affairs to take an overall view of this issue and to develop the process before next year. This would significantly facilitate both the continuous monitoring and management of the investment by the coordination function and the continued assessment by Vårdanalys.

Vårdanalys has also observed certain deficiencies in the information about the terms and conditions of the performance efforts that SALAR provides to the municipalities and county councils, and in the coordination function within the types of consultation by the Ministry of Health and Social Affairs with the National Board of Health and Welfare. However, Vårdanalys considers that there have been positive developments within these areas during 2013.

RECOMMENDATIONS FOR THE CONTINUED DEVELOPMENT OF PRIO 

As the coordination function at the Ministry of Health and Social Affairs has overall responsibility for the continuing management and design of the investment, our recommendations in this progress report are primarily directed there. To a certain extent, the coordination function at SALAR is also included (the final point).

  • The coordination function at the Ministry of Health and Social Affairs should give consideration to the development and clarification of the current target structure in connection with the review of the action plan that is planned for 2014. 

This applies not only to how the targets are formulated but also to how they can, in certain cases, be added to. It should also be reported how overall targets, interim targets and efforts relate to one another.

  • The coordination function at the Ministry of Health and Social Affairs should establish indicators and measurements that make it possible to monitor the development and to evaluate the meeting of targets. 

Decisions about indicators, data sources and target levels do not preclude dynamic management of the investment. These can be reviewed gradually if there is reason to do so. If no decisions are made, there is a risk that it will become unclear where responsibility for the investment lies. The work involved in identifying suitable indicators and measurements, and carrying out data collection, is demanding in terms of time and resources. For this work, Vårdanalys is of the opinion that consideration should be given to strengthening the resources of the coordination function. In the management plan for the investment, the efforts involved in PRIO should also be reported. It should also be reported to which of PRIO’s targets the various efforts are expected to contribute.

  • When a decision is made about efforts to be made, the coordination function at the Ministry of Health and Social Affairs should make it clear which requirements and terms are to apply. 

Such requirements may relate to the production of target descriptions and project plans, as well as matters relating to interim and final reports. It makes it easier for all players involved if the requirements are clear and transparent throughout the entire chain, from the decision on financing through to the final report. One proposal is for reporting to be done in accordance with a set template that makes it clear what has been carried out and achieved in terms of the project plan and target description. It should also be clear how the effort is expected to contribute to one or more of PRIO’s targets, as well as when and how the effort is expected to produce noticeable, lasting effects for people with mental ill health. It is also important to develop procedures for handling deficient or missing reports or results.

  • The coordination function at the Ministry of Health and Social Affairs should continue working on the development of the types of consultation with affected players with regard to the choice, design and assessment of performance-based efforts. 

In view of the scope of the efforts, and the large number of players involved at municipal and county council level, it is of great importance that the national players should have a clear division of responsibility, that they should make sound decisions, and that they should communicate consistent information.

  • The Swedish Government should pass a resolution to the effect that the mission of the coordination functions is extended until 2016, when the investment finishes. 

By monitoring and supporting the development work at local, regional and national level, the coordination functions can make a valuable contribution to the long-term management and development of the investment.