Future Farming Strategy
Mid-term Evaluation Summary Report
Published June 2011
Please contact Julie McGeary (julie.mcgeary@dpi.vic.gov.au) on (03) 5336 6609 for a full version of this report.
The Future Farming Strategy (FFS) aims to develop the capacity of Victorian farm business to become more productive, competitive and sustainable. The strategy includes 20 funded actions from across six government departments. The strategy also identifies 14 further actions that, while not directly funded through the FFS investment, are linked to the aims of the strategy. The Rural and Regional Policy Branch of the Department of Primary Industries (DPI) leads the implementation, reporting and evaluation of the FFS. DPI has a commitment to the Victorian Government to conduct both a mid-term and final evaluation of the FFS.
The purpose of the overall FFS evaluation effort is:
- To demonstrate the influence of the FFS on progress towards improving the productive, competitive and sustainable status of Victorian farm businesses;
- To estimate the long run value of investment in the FFS; and
- To provide opportunities for learning and improvement:
-
- in whole-of-government strategy implementation and delivery; and
- to guide future policy and program development.
Scope of the final evaluation
The final evaluation will demonstrate the contribution the FFS has made towards achievement of expected and unexpected outcomes, and will quantify these contributions where feasible. It will incorporate information and evidence from individual Action evaluations. For return on investment analysis, it will include economic evaluation case studies which have been identified in consultation with DPI’s Economics Branch.
The methodology chosen for the final evaluation will be shaped by the learnings identified in the mid-term evaluation. Therefore, the mid-term evaluation should not be considered a ‘practice run’ for the final evaluation. It has been designed to meet a different purpose and does not include all the elements expected of the final evaluation.
Scope of the mid-term evaluation
The FFS mid-term evaluation provides a mid-point opportunity to review the progress of the FFS implementation. With two more years of the strategy to run, the mid-term evaluation enables any areas for improvement to be identified, and provides all relevant departments with the opportunity to continually improve and strengthen the achievement of outcomes.
The FFS mid-term evaluation examined:
- The extent to which the various funded Actions are being implemented as planned,
- Progress so far for each Action Area against anticipated FFS outcomes,
- Influence of the FFS on strategy and investment,
- Capability of the Actions to evaluate Action performance and achievement of outcomes for the final evaluation.
The mid-term evaluation focused on identifying opportunities for learning and improvement, both in terms of delivery of FFS, and the robustness of the systems and processes (ie the methodology) that will deliver the final evaluation.
Methodology
A mixed methods approach was used for the mid-term evaluation of the FFS. The main methods applied are summarised as follows:
- On-line survey for Action leaders, team members and stakeholders to cover implementation fidelity, progress towards anticipated outcomes, any unintended outcomes, and capability of Action teams to provide evaluative evidence. A response rate of 59.4% was achieved, with 183 respondents participating.
- Twenty follow-up individual interviews to test instances from the survey of implementation, performance and evaluation capability issues to determine success areas that might be emulated or problem areas that need attention.
- Twenty-one individual interviews with senior management (FFS Steering Committee and FFS Interdepartmental Committee) to examine the policy and strategy influence of FFS.
- Feedback workshop with Action Leaders.
The mid-term evaluation assessed progress of the Actions against a number of anticipated FFS outcomes which were identified in the Program logic as follows:
| Short term: delivery |
Improved services, products, facilities, information |
| Clear position on government and industry responsibilities | |
| Improved policy settings | |
| Intermediate: capacity building |
Improved capacity for adoption of new technologies and farm practices |
| Improved capacity for risk management | |
| Improved understanding of government and industry roles and responsibilities | |
| Improved capacity for participation in markets | |
| Improved capacity for better decision making | |
| Medium term practice change | Adoption of new technologies and farm practices |
| Better risk management | |
| Effective participation in markets | |
| Better decision making |
Key findings
Implementation fidelity
The mid-term evaluation assessed the extent to which Actions (or component projects) had been implemented as originally planned. This aspect of evaluation is termed ‘implementation fidelity’ which refers to the degree to which an intervention or program is delivered as intended. By understanding and measuring whether a program has been implemented with fidelity, researchers and practitioners can gain a better understanding of how and why a program works, and the extent to which outcomes can be improved. This is different from measuring progress, which gives an understanding of the extent to which a program is working, rather than how and why it works.
Overall, 28% of respondents reported that Actions had been implemented as planned with no change. Over half identified minor changes, while approximately 14% claimed major changes had been made.
The most common reasons given for implementation change were to take advantage of unexpected opportunities, for continuous improvement purposes, and in response to external factors.
Strong initial project design was cited as being important as it allowed implementation to proceed quickly. However, there was also a high degree of flexibility in the way Actions were implemented. The flexibility of implementation enabled Actions to take advantage of unexpected opportunities as they arose, respond to evolving business priorities, or incorporate learnings gained to shift focus and improve progress towards outcomes. Of those projects that had made major changes, twice as many of the changes resulted in improved progress rather than slowed progress.
Most projects cited external factors as responsible for slowed progress. For example, uncertainty surrounding the Federal Carbon Pollution Reduction Scheme, effects of the Global Financial Crisis, drought, and the reallocation of Action resources to emergency response and recovery activities all had an impact on progress. Very little slowed progress was due to budget or recruitment issues.
Progress against anticipated FFS outcomes
Given the long time frames anticipated for many of the benefits of the FFS investment to be seen, the mid-term evaluation aimed to assess progress towards the achievement of outcomes. The influence of the FFS on strategy and investment was also explored with senior managers. The mid-term evaluation then focused on identifying opportunities for learning and improvement in terms of delivery of FFS.
Action leaders and team members were asked to rate the contribution their Action/project was making towards the anticipated FFS outcomes detailed in the Program logic. Ratings were made against expected progress. Stakeholders were asked to rate the amount of progress they had observed for a sub set of anticipated FFS outcomes.
Short term outcomes: Actions were overwhelmingly progressing ‘as expected’ with respect to providing improved services to agricultural businesses and communities, with 19% of respondents reporting ‘better than expected’ progress.
Stakeholders generally appeared more confident with the amount of progress achieved than team members. More than half of stakeholders reported observing a large amount of progress being made in improved services both to agricultural businesses and agricultural communities.
Overall 43% of Action leaders and team members rated progress ‘as expected’ towards establishing a clear position regarding roles and responsibilities of government and industry. For 28% of respondents this outcome was not considered to be relevant to their Actions.
Progress towards improved policy settings was considered ‘as expected’ or ‘better than expected’ by more than half of Action leaders and team members. Almost a quarter did not consider this outcome relevant to their Actions.
Overall less than 10% of Action leaders and team members reported progress as being ‘less than expected. Similarly less than 7% of stakeholders felt that ‘no progress’ had been observed regarding short term outcomes. This suggests that the FFS is progressing well towards the achievement of short term outcomes. No significant concerns in this area were identified.
Intermediate outcomes: Although the FFS is only mid-way through its implementation, the evaluation identified strong progress towards improved capacity of farm businesses. The most significant of these was ‘improved capacity for better decision making’ which was rated as the intermediate outcome with the most progress by Action leaders/team members and stakeholders alike.
Medium to long term outcomes: The evaluation identified early evidence that farm businesses had actually made changes or adopted new practices at least partially in response to the FFS. This result was unexpected given the long time frames anticipated for many of the benefits of the FFS investment to be seen, and given that the strategy is only mid-way through its implementation.
As per the intermediate outcome results, the most significant of these changes was ‘better decision making’ which was rated as the medium term outcome with the most progress by Action leaders/team members and stakeholders alike.
While a significant proportion of Action leaders, team members and stakeholders reported that it was ‘too early to tell’ (17-28%), ‘no progress observed’ (8-21%) or ‘don’t know’ (23-42%) for the medium term outcomes, this is entirely within the expectations for the strategy delivery timeframes and does not appear to be indicative of any issues or problems with delivery.
A number of factors may be contributing to the early evidence that medium - long term outcomes were being achieved. Extreme seasonal and economic conditions may be driving a rapid pace of change in the Agriculture sector. In recognition of this operating environment, some actions were deliberately designed to meet the needs arising from these conditions and are therefore achieving early success. In addition to this, actions have reported that flexibility in project design and delivery has been a key success factor that has enabled them to respond to changing conditions and provide relevant information and services to farmers.
Other factors identified that may contribute to the early evidence of impact include: stakeholders may not separate FFS specific activities from the broader DPI or government effort; and bias in stakeholder selection for survey participation may have resulted in a more positive response.
Unintended outcomes: Respondents identified 46 unintended outcomes, 74% of which were reported as having a positive effect and 13% a negative effect on progress. The most common ‘unintended outcome’ reported involved improved collaboration, sometimes within the public sector, and sometimes between government and industry.
The most common form of negative outcome centred on uncertainty associated with the implementation of DPI’s ‘Better Services to Farmers’ new service delivery model.
Influence of FFS on strategy and investment
Individual interviews were conducted with members of the FFS Steering Committee and Interdepartmental Committee to assess the influence of the FFS on policy development, strategy and investment decisions.
In their interviews, these Victorian Government senior managers reported the major influences of the FFS. The fact that the FFS provides a stated and clear government position on its commitment to agriculture and priorities for the future was found to be hugely beneficial to the majority of respondents. In particular the FFS has provided:
- Clarity of role of government, including separation of public and private benefit;
- Stability afforded by the FFS four year funding and the commitment to ongoing funding beyond the four year program - particularly for research, and building research and development internal capability;
- Increased ability to leverage private sector co-investment; and
- Increased understanding by industry leaders of the FFS principles, (but less so mainstream farmers).
Examples of where this influence was manifested include the four year Agriculture and Fisheries Group (AFG) strategy, the National Research and Development Framework, the Better Services to Farmers documents, and the Biosecurity Strategy.
The influence of the FFS was rated higher by senior managers from DPI and departments involved with reviewing and making investment decisions (Department of Treasury and Finance, Department of Premier and Cabinet) than other departmental managers (eg Department of Transport, Department of Sustainability and Environment, Department of Human Services, Department of Justice).
Suggestions for improvement
Key factors enabling progress towards achievement of outcomes
Overall, the evaluation showed that the FFS is being well implemented with significant progress towards achieving short, medium and long term outcomes of the strategy. As part of best practice, the mid-term evaluation encouraged Action leaders, team members and stakeholders to identify areas where management and implementation could continue to be improved.
Respondents identified key factors that have been critical in successful project management and delivery. The following major themes emerged, and have been explored for potential improvements:
- Engagement with stakeholders
- Collaboration
- Information
- Project management and flexible delivery
- Interaction between Actions
A range of recommended actions for improvement were identified - either for the remaining implementation of the FFS, or for the development of future whole of government strategies, and are listed at the end of this summary.
Evaluation
The mid-term evaluation also tested the robustness of the systems and processes that will deliver the final evaluation.
To this extent, the mid-term evaluation assessed the strength of evaluation evidence to date, and examined the evaluation capability needs within Actions so that the final evaluation can be completed in the longer term. Action leaders and team members were asked to rate the strength of evaluation evidence they had to support the claims of progress they were making. They were also asked what assistance they required to strengthen their evaluation evidence.
Overall, Action leaders and team members reported that the strength of evaluation evidence was medium to high. All Action teams were aware that evaluation is required and have evaluation plans in place. As expected, strength of evaluation evidence was rated stronger for the shorter term outcomes and became weaker or less certain for the medium to longer term outcomes.
It was clear however that several Action teams had not previously considered their evaluation (and evaluation evidence) in terms of the FFS outcomes tested in this mid-term evaluation. As a result, several found the evaluation component of the survey difficult to complete.
In order to complete a ‘whole of strategy’ evaluation, it will be important to increase alignment between Action level evaluations and the FFS Program logic and in turn, improve the ability of Actions to provide evidence of impact against the FFS outcomes.
Almost 80% of team members reported they were collecting baseline measurements at the Action level. The need for evaluation assistance was reported by 22% of team members and a further 24% were not sure if they need help or not. Emerging from the mid-term evaluation will be a need to address identified gaps in capability. Several recommendations have been identified and have been listed below.
Opportunities for improving the evaluation methodology of the FFS
As with any evaluation, the process of collecting and analysing data identified opportunities to improve the methods used for the final evaluation.
Analysis of the methodology identified areas that were difficult or confusing for survey participants, or where there were limitations within the survey methodology.
Action leaders were asked to nominate potential survey participants. This process was inconsistent and may have led to some positive bias in the sample.
Stakeholders may not directly relate their involvement to the FFS brand, and there could be a variable connectedness and depth of understanding. The stakeholder survey was designed with this risk taken into account. Nevertheless, nominated participants, particularly stakeholders, displayed disparate levels of understanding of the FFS and anticipated outcomes at the whole-of-strategy level. With a total of 183 survey responses and over 40 interviews, however, the effect of this variable understanding should be minimised.
The survey tested progress against a range of short and medium term outcomes. Differences between questions were necessarily subtle and potentially confusing or difficult to answer. This made survey completion challenging for participants, and interpretation of results was therefore problematic.
Recommended actions for addressing these limitations have been identified and are listed below.
Next steps
Suggestions for improvement emerging from the mid-term evaluation have been incorporated into the recommendations made in the full report, which are summarised below with responsibilities for action.
| Recommendation | Responsible person | |
|---|---|---|
| 1 | Continue to develop and maintain strong relationships with stakeholders and collaborators throughout the life of FFS projects. | FFS Action leaders, Steering Committee and IDC members |
| 2 | For future strategies, ensure relationships with stakeholders and collaborators are established early in the project development stage, and then fostered continuously for the life of projects. | Future strategy developers and relevant committee members |
| 3 | For future strategies, ensure data and information sharing processes are formally agreed between collaborating parties during the project development stage. | Future strategy developers and relevant committee members |
| 4 | Encourage continual interagency dialogue to improve consistency of messages and coordination of services delivered to common target audiences. | FFS Action leaders, Steering Committee and IDC members |
| 5 | Review and update if necessary, internal departmental communications procedures to ensure they enable rapid approval processes and provide adequate support for the production of FFS branded materials. | Manager FFS Implementation and Reporting |
| 6 | Review and update FFS variation, project reporting (including final reporting) and budget approval processes to ensure efficient completion of actions and the FFS. | Manager FFS Implementation and Reporting |
| 7 | For future strategies, ensure such governance processes identified in Recommendation 6 are in place well in advance of implementation. | Future strategy developers and relevant committee members |
| 8 | Convene an annual meeting of FFS Action leaders and team members to enable improved collaboration and understanding of each other’s achievements. | Manager FFS Implementation and Reporting |
| 9 | Work with Actions to ensure specific evaluation support requests are addressed through the provision of advice, direction to expertise or other, with the understanding that it is not the role of the FFS Evaluation Manager to undertake evaluations on behalf of the Actions. | Manager FFS Evaluation |
| 10 | Work with existing DPI Evaluation capability (Strategic Evaluation Unit and the Agriculture and Fisheries Evaluation Reference Group) to examine how overall evaluation capability can be improved and any gaps in capability addressed. | Manager FFS Evaluation |
| 11 | Work with Action leaders to explore alternative survey design and structure which is still focused on outcomes but not so dependent on familiarity with the FFS Program logic model. | Manager FFS Evaluation |
| 12 | Work with each Action leader to review Action level evaluation plans and Program logic to improve alignment of data collection and outcomes with the whole-of-strategy evaluation effort. | Manager FFS Evaluation |
