Skip to main content Start of content

PACP Committee Report

If you have any questions or comments regarding the accessibility of this publication, please contact us at accessible@parl.gc.ca.

Government Response to the Seventeenth Report of the Standing Committee on Public Accounts: Chapter 1, “Evaluating the Effectiveness of Programs,” of the Fall 2009 Report of the Auditor General of Canada

The Government agrees with the Standing Committee on Public Accounts (PAC) on the importance of evaluating programs to provide reliable information on program effectiveness for decision making, and values the Committee’s support for the continued development of effectiveness evaluation in the federal government and for the recently-strengthened policy requirements for evaluation.

The Government recognizes program evaluation as a key source of information on program value for money, including program effectiveness, efficiency, economy and relevance. As a management tool, evaluation supports policy and program improvement, expenditure management, Cabinet decision making and public reporting by providing reliable, neutral assessments of the value for money of government programs.

With the renewal of the Government of Canada’s Expenditure Management System in 2007 and in keeping with the Government Response to PAC’s Fourth Report, tabled in July 2008, which accepted a recommendation that the Treasury Board of Canada Secretariat reinforce the importance of evaluation by adding program evaluation as a key requirement in the Expenditure Management System, evaluation plays an increasingly important role in the effective management of public spending. To support that role, Treasury Board’s 2009 Policy on Evaluation introduced significant changes for the evaluation function in departments and across the Government of Canada, which aim to expand evaluation coverage of programs, to focus evaluations on value for money issues, to ensure the collection of ongoing performance data to support evaluations, and to improve the governance and neutrality of the function.

Because of the substantial changes introduced in the 2009 Policy on Evaluation, a four-year gradual implementation has been adopted to give departments adequate time to adapt to the new requirements and to build their evaluation capacity to meet the requirements of the policy.

During the implementation of the Policy on Evaluation and the Policy on Management, Resources and Results Structures, TBS will continue to provide advice and support to departments on improving their evaluation functions and their ongoing performance measurement. The annual Health of the Evaluation Function Report will be publicly released after it has been approved by Treasury Board, and will track progress in the evaluation function.

RECOMMENDATION 1

That Canadian Heritage and Human Resources and Skills Development Canada provide the Public Accounts Committee with action plans by 31 December 2010 of how they intend to address the Office of the Auditor General’s recommendations included in Chapter 1 of the Fall 2009 Report.

The recommendation is accepted.

In its response to the 14th Report of the Standing Committee on Public Accounts on Chapter 2 of the December 2008 Report of the Auditor General of Canada on the Governance of Small Federal Entities, the Government noted that departments and agencies will continue to be responsive to the Committee’s requests for information and to take all reasonable steps to ensure that the Committee has the information needed when conducting hearings. In this context, Government departments and agencies continue to provide the Committee with all appropriate information when called to appear before the Committee.

In anticipation of appearing before the Committee on Chapter 1 of the Fall 2009 Auditor General Report entitled “Evaluating the Effectiveness of Programs”, all departments implicated in the Chapter prepared action plans. However, Canadian Heritage and Human Resources and Skills Development Canada were not called to appear before the Committee and therefore did not provide them to the Committee.

In the interests of providing the Committee with existing documents that could assist Committee members in gaining a greater understanding of the Government’s efforts in this area, the Government is pleased to provide additional information to complement that already provided to the Committee. As such, the departmental action plans prepared by Canadian Heritage and Human Resources and Skills Development Canada are attached as annexes to the Government Response.

In summary, these two departmental action plans, along with the other action plans already received by the Committee, demonstrate the Government’s commitment to improve performance information, to strengthen evaluation functions across government and to improve its ability to manage for results.

Recommendation 2

That beginning in 2010-11 the Treasury Board of Canada Secretariat prepare an annual report on the status of the evaluation function and make this report available publicly.

The recommendation is accepted.

TBS will prepare its annual Health of the Evaluation Function Report, as required under the 2009 Policy on Evaluation, and make this report available publicly on the Treasury Board of Canada website after it has been approved by Treasury Board. This report will contain information on the status of various aspects of the Government of Canada evaluation function, including its progress in implementing the 2009 Policy on Evaluation.

The report will widely communicate the state of the function as monitored by TBS, inform and communicate TBS activities to support the government-wide function, support deputy heads and other stakeholders by proposing approaches to improve the federal evaluation function, and thus serve as an information tool for parliamentarians and Canadians in tracking progress across the function.

Recommendation 4

That Environment Canada, Fisheries and Oceans Canada and Human Resources and Skills Development Canada provide the Public Accounts Committee with a strategy by 31 January 2011 of how they intend to improve their performance measurement frameworks.

The recommendation is accepted and the requested information is provided as follows.

The Treasury Board renewed the Policy on Management, Resources and Results Structures (MRRS) in February 2010. Originally introduced in April 2005, the Policy provides a common approach to the collection, management and reporting of financial and non-financial performance information.

In essence, the Policy on MRRS seeks to link resources (planned and actual) to intended results for all program expenditures and works to enhance decision-making by providing access to accurate financial and non-financial performance information.

Departments subject to the Policy are required to develop an MRRS, which is composed of the following components, which are used as the basis for their reports to Parliament:

  • Strategic Outcomes (SO): measurable long-term enduring benefits to Canadians;
  • Program Activity Architecture (PAA): an inventory of programs linked to strategic outcomes, including a supporting performance measurement framework (PMF), which sets out the expected results to be achieved and the specific outputs to be produced by a program along with performance measures; and
  • Governance Structure: the decision-making mechanisms, responsibilities and accountabilities within a department.

The performance measurement framework is a key component of the Policy on MRRS. Organizations must identify performance measures for each strategic outcome and program identified in their PAA.

The Government recognizes that a strong performance measurement framework is an important factor in facilitating evaluations. As required by the 2009 Policy on Evaluation, heads of evaluation of all large departments and agencies review and provide advice on departmental performance measurement frameworks and strategies, and will also prepare annual reports for submission to their departmental evaluation committees on the state of performance measurement within their departments, in terms of how well it supports evaluation. These reports will help departments identify needed improvements in performance measurement to support the conduct of evaluations.

The development of performance information is an ongoing process and significant progress has been made in all three departments since the audit was conducted. The three departments continue to be engaged and to work with the Treasury Board of Canada Secretariat to improve their performance measurement frameworks and related performance measurement information in order to better support management and decision making.

As highlighted below, Environment Canada, Fisheries and Oceans Canada and Human Resources and Skills Development Canada have all made modifications since the audit to further improve and strengthen the quality of their PMFs.

Fisheries and Oceans Canada

Fisheries and Oceans Canada has undertaken a complete redesign of its Management, Resources and Results Structure (MRRS) for 2011-12, of which a key component is its Performance Measurement Framework (PMF).

Fisheries and Oceans Canada has made considerable efforts to increase the level of precision and quality of performance indicators so that it provides a more effective and reliable means by which performance may be assessed and reported to Canadians.

Further, Fisheries and Oceans’ performance indicators for 2011-12 have been developed with input from the departmental Evaluation Directorate to ensure they will support upcoming evaluations, and will be subjected to periodic internal and external reviews to ensure continuous improvement.

Fisheries and Oceans Canada has also launched a Performance Measurement Action Plan, which is a comprehensive phased approach to ensure that Performance Measurement Strategies are developed and implemented for all departmental programs by December 2011. These Performance Measurement Strategies will support both the Performance Measurement Framework of the department as well as individual evaluations.

Environment Canada

Since the OAG audit was conducted, Environment Canada undertook a major revision to its Program Activity Architecture (PAA) and Performance Measurement Framework (PMF). The PMF in effect for the 2010-11 fiscal year includes coverage of all PAA programs.

Performance measurement strategies are being renewed or developed for all PAA programs over several years and on a phased basis, with the timing of these strategies aligned to the schedule of program evaluations.

An inventory of performance indicators currently in use department-wide over and above those in the PMF is being developed; these will provide a baseline against which proposed changes to indicators may be assessed, and this work will enable the identification of important gaps that need to be addressed.

As programs proceed to implement systems to collect and analyze performance data, areas where revisions are needed to ensure relevance, reliability and utility of reported information are identified through a process of regular review and challenge.

Environment Canada’s Audit and Evaluation Branch will review the implementation of the PMF and program performance measurement strategies through several mechanisms. For example, pre-evaluation evaluability assessments will be conducted two years prior to each planned evaluation and will identify possible issues with the availability or quality of performance information as well as other issues related to evaluability. Implementation of management responses to recommendations in evaluations, particularly those dealing with performance measurement, will continue to be monitored regularly.

Human Resources Skills Development Canada

In 2009, HRSDC undertook a broad-based review of its Departmental Performance Measurement Framework (PMF) using Assistant Deputy Minister Strategic Outcome Planning Tables as the forum for discussion. This resulted in a strengthened 2010-11 PMF with improved expected results statements and performance indicators.

As part of its annual MRRS update, HRSDC will be looking to further improve specific indicators within the PMF for 2011-12, following a review by senior management and the Departmental Audit Committee. As with previous years, the Evaluation Directorate provided input to the development of the 2011-12 departmental performance indicators. In preparation for 2012-13, the Department will undergo a broader review of the departmental PMF.

HRSDC supports evidence-based policy and program development, diagnostic exercises, forecasting and evaluation, as well as the collection and analysis of data in support of improving indicators and data reliability. In support of this function and to strengthen management of PMF-related information, HRSDC is currently developing a web-based application for storing and sharing performance measurement information and data. This central repository of current and historical information on the PMF will enable both increased reporting flexibility and comparability of performance data across several years.

Recommendations 3, 5, 6, 7, 8

Recommendation 3

That the Treasury Board of Canada Secretariat include in the annual report on the status of the evaluation function, an update on its oversight and support activities for effectiveness evaluation.

Recommendation 5

That the Treasury Board of Canada Secretariat include in the annual report on the status of the evaluation function, information on the number of effectiveness evaluations, and in which departments, that had difficulty assessing program success due to insufficient performance information.

Recommendation 6

That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function, information on the percentage of direct program spending and grants and contributions programs covered by evaluations in the fiscal year, as well as the percentage evaluated over the five year cycle.

Recommendation 7

That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function, information on the number of evaluators employed in large departments in the federal government.

Recommendation 8

That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function information on the percentage of evaluations that are conducted by contractors.

The recommendations are accepted, and a clarification on roles and responsibilities is provided. 

As defined under the 2009 Policy on Evaluation, evaluations are required to assess the value for money of programs, including the extent of program effectiveness, efficiency, economy and relevance. As indicated in the Government Response to recommendation 2, TBS will prepare an annual Health of the Evaluation Function Report, as required under the 2009 Policy on Evaluation. The report will contain information about the status of the evaluation function, its progress in meeting legislated requirements for evaluation of ongoing programs of grants and contributions under the Financial Administration Act, as well as its progress in implementing the 2009 Policy on Evaluation. The Health of the Evaluation Function Report will also contain information identified in PAC’s recommendations, including information on:

  • TBS’ oversight and support activities for the evaluation function;
  • the number of evaluations TBS has received from departments;
  • the percentage of direct program spending and grants and contribution programs covered by evaluations in the reporting year, as well as the percentage evaluated over the five-year cycle;
  • the number of evaluators employed in large departments in the federal government;
  • the use of contractors in the conduct of evaluations; and
  • the percentage of large departments assessed under the Management Accountability Framework whose evaluations cited difficulties in assessing program effectiveness due to insufficient performance information.

It is the responsibility of individual departments to report instances where their evaluations had difficulty assessing program success due to insufficient performance information, as under the Standard on Evaluation for the Government of Canada, every departmental evaluation report must disclose “the constraints of the evaluation with respect to the design, methods and consultations.” Further, heads of evaluation in large departments and agencies are required by the Directive on the Evaluation Function to submit to their Departmental Evaluation Committee an annual report on the “state of performance measurement of programs in support of evaluation”. 

Recommendation 9

The Office of the Auditor General conduct a follow-up audit within five years of the initial audit on whether departments are able to meet the requirements for effectiveness evaluation, and whether they are conducting evaluations in an economical and efficient manner.

The Government Response does not address the recommendations that fall outside its area of responsibility. As such, the Government is unable to respond to the Committee’s recommendation 9, which is directed for consideration of the Office of the Auditor General. The Auditor General of Canada has authority to respond to this recommendation independently in her capacity as an agent of Parliament


DEPARTMENT OF CANADIAN HERITAGE

Action Plan for the Implementation of the Auditor General’s Recommendations

on “Evaluating the Effectiveness of Programs”

RECOMMENDATION

DEPARTMENTAL RESPONSE

ACTION PLAN / STATUS

1.37 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should develop and implement action plans to ensure that ongoing program performance information is collected to support effectiveness evaluation.

AGREED

1. The Department is executing its Action Plan for Implementation of the Management, Resources and Results Structures (MRRS) Policy to ensure that program staff are able to fulfill their responsibility for developing and maintaining performance measurement strategies. Action plan measures include:

­ the provision of information sessions and workshops,

­ the establishment of indicators and corresponding targets,

­ the development of robust methodologies to demonstrate outcomes,

­ the establishment of a centre of expertise on performance measurement within the Department,

­ the design and implementation of adequate tools and guidelines,

­ the establishment of relevant information technology and systems, and

­ the regular analysis and reporting of collected data.

This action plan is expected to be completed by end of the 2011–12 fiscal year.

PARTIALLY IMPLEMENTED

IN PROGRESS

Adjustments were made to the MRRS Implementation Action Plan due to departmental budget reductions and the launch of the Transformation Initiative. The Action Plan will be updated once the Transformation Initiative and the implementation of the Internal Services Review, in particular, are concluded.

In the meantime, the department continues to make progress in improving its state of performance measurement.

Provision of Information Sessions and Workshops: In August 2009, the Deputy Minister of PCH identified performance measurement as a learning priority. In September 2009, the Strategic Planning Directorate (SPD) provided performance measurement information and workshop sessions to all Program Managers, Program Directors and Program Officers. Human Resources Intranet has listed training on Performance Measurement as a learning priority for 2010-2012 for all PCH employees. SPD is therefore engaged in delivering two performance measurement training sessions in November, 2010. More in-depth training may be offered during Spring 2011. In the meantime, SPD is working with the Centre of Expertise (CoE) on


RECOMMENDATION

DEPARTMENTAL RESPONSE

ACTION PLAN / STATUS

   

­ Grants and Contributions, Evaluation Services Directorate (ESD) and Financial Management Branch (FMB) towards the adoption of a common terminology with regards to performance management throughout the Department. SPD is also giving advice to branches and programs on performance measurement strategies and their alignment with the Performance Measurement Framework (PMF).

­ Establishment of Indicators and Corresponding Targets: PCH continues to improve its Performance Measurement Framework. The PMF has continued to mature and evolve. The 2011-2012 PMF contains improved indicators and an increased number of targets. Pending Transformation Initiative organizational changes, the revised action plan will outline next steps and scope of modifications to the PMF for the coming months.

­ Development of Robust Methodologies to Demonstrate Outcomes: PCH undertook a PMF methodology development exercise in summer 2009 that resulted in the refinement of indicators and expected results at SO and PA levels. In addition, targets have been identified for most of the PA and PSA.

Establishment of a Centre of Expertise on Performance Measurement within the Department: PCH has (staffed) a Manager and an analyst to the performance measurement file. The establishment of a Centre of Expertise is on hold due to current fiscal restraints and pending further decisions on delivery models of corporate functions. Those decisions are expected to be implemented in the beginning of 2011. Additionally, the SPD has implemented a formal planning network, notably through the development of a



RECOMMENDATION

DEPARTMENTAL RESPONSE

ACTION PLAN / STATUS

   

­ Level IV (Director- and Manager-level) Planning Working Group, which supports the Level III Integrated Planning Committee in the governance structure.

­ Design and Implementation of Adequate Tools and Guidelines: PCH has developed methodology fiches to elaborate on expected results and indicators at the SO and PA level. These were very helpful and used for reporting requirements like the 2009-2010 DPR. Further development of the sub-sub-activity level is discretionary and has been delayed due to other priorities; however, it will be revisited at a later date.

­ Establishment of Relevant Information Technology and Systems: This project was cancelled due to fiscal restraints. Further direction will be given once the results of the Transformation Initiative have been implemented. Details will then unfold.

­ Regular Analysis and Reporting of Collected Data: PCH has been piloting the data collection process within the Heritage Group (Program Activity 3) for two fiscal years, using 2008-2009 as a reference year. The 2009-2010 data collection contributed to the improvement of the reporting capacity.

­ PCH will prepare to expand the data collection, analysis and reporting during FY2011-2012.

 

2. The Department’s Office of the Chief Audit and Evaluation Executive is continually providing advice and support to Department managers in their efforts to implement this action plan. In line with the Transfer Payment Policy, this office is also providing timely advice and support on program design and performance measurement strategies

COMPLETED – PCH’s OCAEE will continue to systematically review and provide sign-off on all TB Submissions, ensuring the quality and validity of information provided on evaluations and performance measurement strategies. Evaluation managers will also continue to review logic models, performance indicators and evaluation strategies for programs in their portfolio. This structure ensures coverage of the entire departmental



RECOMMENDATION

DEPARTMENTAL RESPONSE

ACTION PLAN / STATUS

 

through the review of official approval documents for the creation and renewal of new or existing programs.

Program Activity Architecture and enables better monitoring of the quality of performance measurement strategies throughout the department.

 

3. Finally, as required under the new Evaluation Policy, the office will be submitting, in the 2010–11 fiscal year, its first annual report to the departmental evaluation committee on the state of performance measurement in the Department.

PARTIALLY IMPLEMENTED

Phase I – Preliminary Study

The Chief Audit and Evaluation Executive completed a preliminary assessment on the state of performance measurement of programs at PCH in support of evaluation. The main objective was to assess the appropriateness and availability of data collected by programs for effectiveness evaluations. Broad results were presented to the Departmental Evaluation Committee late June 2010.

Phase II – Study

This phase corresponds to the actual conduct of the assessment. Some of the issues to be addressed include the following:

­ portrait of results data availability and collection methods;

­ actions taken by program management and corporate functions to implement and improve performance measurement;

­ factors affecting the implementation of a performance culture at PCH, including support offered by PCH senior management;

­ training offered to program staff and training required;

­ clarity of the roles and responsibilities in the Department for ensuring the adequate implementation of performance measurement strategies.

Deliverable: A final report including:

­ assessment of the state of performance measurement at PCH in support to evaluation;

recommendations and suggested methodologies to senior management and programs to improve and strengthen performance measurement within PCH.

Timeline: March 2011.




RECOMMENDATION

DEPARTMENTAL RESPONSE

ACTION PLAN / STATUS

1.43 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should consider the merits of including external experts on their departmental evaluation committees. The Treasury Board of Canada Secretariat should provide guidance to departments in this regard.

AGREED

At the Department, oversight of the evaluation function is provided by the Strategic Policy, Planning and Evaluation Committee (SPPEC), chaired by the Deputy Head. This structure provides opportunities for enhanced integration between the policy, planning, and evaluation functions of the department. As well, the Department already brings key evaluation reports when necessary to its department audit committee for review and discussion. The Department will collaborate with the Treasury Board of Canada Secretariat’s Centre of Excellence in Evaluation to assess the value added of integrating the advice of external evaluation experts to inform the work of departmental evaluation committees.

COMPLETED

1. The Department’s Chief Audit and Evaluation Executive will continue, in future, to bring all key evaluation reports to its departmental audit committee for review and discussion.

2. PCH already collaborates with Treasury Board of Canada Secretariat’s Centre of Excellence for Evaluation (CEE) on several evaluation related issues. The Department is well represented and actively contributes to all interdepartmental working groups put in place by the CEE for the implementation of the 2009 Policy on Evaluation. PCH participated to CEE’s consultation and discussions with departments to clarify the role of external evaluation experts within departmental evaluation committees. PCH provided the CEE with comments on the challenges related to the implementation of this recommendation at PCH, comments that helped develop guidance on this issue.

1.48 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should implement systematic processes to determine whether their effectiveness evaluations are meeting government-wide requirements and internal corporate needs, and act on areas identified for improvement.

AGREED

The Department recognizes the need to establish systematic processes to assess whether effectiveness evaluations are addressing needs. The Evaluation Services Directorate is already developing a performance measurement framework and management strategy to identify clear performance expectations and standards for the Department’s evaluation function. A systematic process to collect, analyze, and report on performance data and client satisfaction will be implemented in order to identify

PARTIALLY IMPLEMENTED

Meeting government-wide requirements:

COMPLETED – 1.The first draft of the final report for each evaluation conducted by PCH Evaluation Services Directorate is peer-review internally. A standardized tool is used systematically to assess different criteria and capture relevant comments in accordance with the Standards of Evaluation for the Government of Canada and Treasury Board Secretariat’s MAF criteria for the quality of evaluations.


RECOMMENDATION

DEPARTMENTAL RESPONSE

ACTION PLAN / STATUS

The Treasury Board of Canada Secretariat should monitor and provide any additional support it considers necessary for the implementation of these processes.

areas of improvements. Based on data collected during the first year of implementation (the 2010–11 fiscal year), periodic reporting to the Departmental Evaluation Committee should begin by the 2011–12 fiscal year. This performance information will complement data already reported in the context of the annual management accountability framework assessments conducted by the Treasury Board of Canada Secretariat.

Meeting internal corporate needs and act on areas identified for improvement:

COMPLETED – 1. Program managers, DGs, ADMs and the Deputy Minister are always consulted on the 5-year Departmental Evaluation Plan to ensure that programming adequately address specific reporting needs. Terms of reference and evaluation questions for high risk programs are discussed and approved by the departmental evaluation committee chaired by the Deputy Minister.

IN PROGRESS – 2. The Evaluation Services Directorate has reviewed its own logic model and developed a performance measurement framework to monitor and assess progress towards achieving the expected results. Clear performance expectations have been identified and are part of the Department’s Office of the Chief Audit and Evaluation Executive 2010-11 / 2012-13 Integrated Business Plan. Fiscal year 2010-2011 will be dedicated to develop a baseline of data, 2011-2012 to implement the data collection system and by April 1 2012, the Evaluation Services Directorate will review and update its service charter and inform its departmental clients. Once implemented, service delivery will be monitored to assess compliance and make necessary adjustments.

IN PROGRESS – 3.

­ The Evaluation Services Directorate developed and piloted a client survey questionnaire to collect client satisfaction data after the conduct of an evaluation. The questionnaire was updated and programmed to be accessed online. Effective January 2011, this tool will systematically be used to collect performance data and identify areas of improvement.

Also, effective in January 2011, the Evaluation Services Directorate will systematically hold and document


RECOMMENDATION

DEPARTMENTAL RESPONSE

ACTION PLAN / STATUS

   

lessons learned sessions with program management after the completion of an evaluation. The goal of these sessions will be to review the evaluation process, assess how well program management’s needs were met, highlight best practices and identify possible improvements.

­ The Evaluation Services Directorate is developing an annual client survey to collect additional data on the quality and timeliness of consulting services/advices, as well as usage of evaluation results. This tool will be implemented in the fiscal year 2011-2012.


Department of Human Resources and Skills Development Canada

Action Plan for the Implementation of the Auditor General’s Recommendations

on “Evaluating the Effectiveness of Programs”

AUDITOR GENERAL OF CANADA RECOMMENDATIONS

MANAGEMENT RESPONSE

MANAGEMENT PLAN

ACTION(S) TO BE UNDERTAKEN

COMPLETION DATE

RESPONSIBILITY

AND/OR RC

NUMBER

RECOMMENDATION 1.37 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should develop and implement action plans to ensure that ongoing program performance information is collected to support effectiveness evaluation.

Human Resources and Skills Development Canada’s response.

Agreed. The Department accepts this recommendation that echoes the intent of various Treasury Board of Canada Secretariat policies, including the 2009 Evaluation Policy. As such, the Department already gathers and monitors ongoing performance information to support effectiveness evaluation, including:

• monitoring implementation of management responses to previous evaluations that have identified data issues for current programs;

• undertaking early evaluative work in advance of the formal initiation of effectiveness evaluations as part of the evaluation planning process, to review the state of performance data and logic models; and

Develop and submit an annual report on the state of performance measurement, as required by the Policy on Evaluation.

Management response follow-up study (2004-2005 through 2006-2007) was submitted to the Departmental Evaluation Committee (DEC) on July 7, 2009. Frequency and approach to future monitoring to be discussed at DEC.

Ongoing

March 2010

(annual updates)

December 8, 2009

(periodic reporting)

Ongoing

DG Evaluation,

Strategic Policy and Research Branch

DG Evaluation,

Strategic Policy and Research Branch

DG Evaluation,

Strategic Policy and Research Branch


AUDITOR GENERAL OF CANADA RECOMMENDATIONS

MANAGEMENT RESPONSE

MANAGEMENT PLAN

ACTION(S) TO BE UNDERTAKEN

COMPLETION DATE

RESPONSIBILITY

AND/OR RC

NUMBER

 

• monitoring the implementation of new programs (for example, Economic Action Plan initiatives) to ensure the necessary administrative and performance data are available to support future evaluation activities.

The Department is also undertaking a comprehensive review, refinement, and validation of its Performance Measurement Framework (PMF) to make it more robust and comprehensive, to support ongoing planning, monitoring, and managing for results and the Evaluation Directorate has been actively involved in this work. The progress made on the departmental PMF will support both performance monitoring as well as evaluation of relevance and effectiveness.

Ongoing

Complete a comprehensive review, refinement, and validation of the Performance Measurement Framework (PMF) with engagement by ADMs across the Department.

Work with Treasury Board Secretariat to continue to strengthen PMF on an ongoing basis.

Ongoing

September, 2009 (complete)

Annual updates

DG Evaluation,

Strategic Policy and Research Branch

DG Corporate Planning and Accountability, Strategic Policy and Research Branch


AUDITOR GENERAL OF CANADA RECOMMENDATIONS

MANAGEMENT RESPONSE

MANAGEMENT PLAN

ACTION(S) TO BE UNDERTAKEN

COMPLETION DATE

RESPONSIBILITY

AND/OR RC

NUMBER

RECOMMENDATION. 1.43 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should consider the merits

of including external experts on their departmental evaluation committees. The Treasury Board of Canada Secretariat should provide guidance to departments in this regard.

Human Resources and Skills Development Canada’s response.

Agreed. The Department has, in the past, included members from outside the Department on its departmental evaluation committee. The Department will reconsider the formal inclusion of external experts on the current Departmental Evaluation Committee and will look to the Treasury Board of Canada Secretariat for guidance on this part of the Evaluation Policy.

Discuss external membership at the Departmental Evaluation Committee, and revise terms of reference as required.

November 17, 2009

DG Evaluation,

Strategic Policy and Research Branch


AUDITOR GENERAL OF CANADA RECOMMENDATIONS

MANAGEMENT RESPONSE

MANAGEMENT PLAN

ACTION(S) TO BE UNDERTAKEN

COMPLETION DATE

RESPONSIBILITY

AND/OR RC

NUMBER

RECOMMENDATION 1.48

Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development

Canada should implement systematic processes to determine whether their effectiveness evaluations are meeting government-wide requirements and internal corporate needs, and act on areas identified for improvement. The Treasury Board of Canada Secretariat should monitor and provide any additional support it considers necessary for the implementation of these processes.

Human Resources and Skills Development Canada’s response.

Agreed. The Department currently employs a variety of systematic processes to ensure the quality and relevance of its evaluations for both internal and government-wide needs. External peer reviewers and evaluation advisory committees are a mandatory element of the evaluation work to ensure that evaluations are meeting information needs. Further work in systematically determining whether evaluations are meeting government and senior management needs will be developed, building upon the current annual Management Accountability Framework assessment process led by the Treasury Board of Canada Secretariat. Areas for improvement will be identified and reported to the Departmental Evaluation Committee.

Develop the Departmental Evaluation Plan, in consultation with senior management, to identify the appropriate level, focus and sequencing of evaluation resources as well as areas for improvement.

Identify possible areas of improvement through the annual Management Accountability Framework assessment. Present and discuss at the Departmental Evaluation Committee.

April 2010

(annual updates)

April / May 2010

(annual process)

DG Evaluation,

Strategic Policy and Research Branch