|
The
Government agrees with the Standing Committee on Public Accounts (PAC) on the
importance of evaluating programs to provide reliable information on program
effectiveness for decision making, and values the Committee’s support for the
continued development of effectiveness evaluation in the federal government and
for the recently-strengthened policy requirements for evaluation.
The Government recognizes program
evaluation as a key source of information on program value for money, including
program effectiveness, efficiency, economy and relevance. As a management
tool, evaluation supports policy and program improvement, expenditure
management, Cabinet decision making and public reporting by providing reliable,
neutral assessments of the value for money of government programs.
With the
renewal of the Government of Canada’s Expenditure Management System in 2007 and
in keeping with the Government Response to PAC’s Fourth Report, tabled in July
2008, which accepted a recommendation that the Treasury Board of Canada
Secretariat reinforce the importance of evaluation by adding program evaluation
as a key requirement in the Expenditure Management System, evaluation plays an
increasingly important role in the effective management of public spending. To
support that role, Treasury Board’s 2009 Policy on Evaluation introduced
significant changes for the evaluation function in departments and across the
Government of Canada, which aim to expand evaluation coverage of programs, to
focus evaluations on value for money issues, to ensure the collection of
ongoing performance data to support evaluations, and to improve the governance
and neutrality of the function.
Because of
the substantial changes introduced in the 2009 Policy on Evaluation, a
four-year gradual implementation has been adopted to give departments adequate
time to adapt to the new requirements and to build their evaluation capacity to
meet the requirements of the policy.
During the
implementation of the Policy on Evaluation and the Policy on
Management, Resources and Results Structures, TBS will continue to provide
advice and support to departments on improving their evaluation functions and
their ongoing performance measurement. The annual Health of the Evaluation
Function Report will be publicly released after it has been approved by
Treasury Board, and will track progress in the evaluation function.
RECOMMENDATION
1
That Canadian Heritage and Human Resources and
Skills Development Canada provide the Public Accounts Committee with action
plans by 31 December 2010 of how they intend to address the Office of the
Auditor General’s recommendations included in Chapter 1 of the Fall 2009
Report.
The recommendation is accepted.
In its
response to the 14th Report of the Standing Committee on Public
Accounts on Chapter 2 of the December 2008 Report of the Auditor General of
Canada on the Governance of Small Federal Entities, the Government noted that
departments and agencies will continue to be responsive to the Committee’s
requests for information and to take all reasonable steps to ensure that the
Committee has the information needed when conducting hearings. In this context,
Government departments and agencies continue to provide the Committee with all
appropriate information when called to appear before the Committee.
In
anticipation of appearing before the Committee on Chapter 1 of the Fall 2009
Auditor General Report entitled “Evaluating the Effectiveness of Programs”, all
departments implicated in the Chapter prepared action plans. However, Canadian
Heritage and Human Resources and Skills Development Canada were not called to
appear before the Committee and therefore did not provide them to the
Committee.
In
the interests of providing the Committee with existing documents that could
assist Committee members in gaining a greater understanding of the Government’s
efforts in this area, the Government is pleased to provide additional
information to complement that already provided to the Committee. As such, the
departmental action plans prepared by Canadian Heritage and Human Resources and
Skills Development Canada are attached as annexes to the Government Response.
In
summary, these two departmental action plans, along with the other action plans
already received by the Committee, demonstrate the Government’s commitment to
improve performance information, to strengthen evaluation functions across
government and to improve its ability to manage for results.
Recommendation 2
That beginning in 2010-11 the Treasury Board of Canada Secretariat prepare an annual report on the status of the evaluation function and
make this report available publicly.
The
recommendation is accepted.
TBS
will prepare its annual Health of the Evaluation Function Report, as required
under the 2009 Policy on Evaluation, and make this report available
publicly on the Treasury Board of Canada website after it has been approved by
Treasury Board. This report will contain information on the status of various
aspects of the Government of Canada evaluation function, including its progress
in implementing the 2009 Policy on Evaluation.
The
report will widely communicate the state of the function as monitored by TBS,
inform and communicate TBS activities to support the government-wide function,
support deputy heads and other stakeholders by proposing approaches to improve
the federal evaluation function, and thus serve as an information tool for
parliamentarians and Canadians in tracking progress across the function.
Recommendation 4
That Environment Canada, Fisheries and Oceans Canada and Human Resources and Skills Development Canada provide the Public Accounts Committee with
a strategy by 31 January 2011 of how they intend to improve their performance
measurement frameworks.
The
recommendation is accepted and the requested information is provided as
follows.
The Treasury Board
renewed the Policy on Management, Resources and Results Structures (MRRS) in February 2010. Originally introduced in April 2005, the Policy provides
a common approach to the collection, management and reporting of financial and
non-financial performance information.
In essence, the Policy on MRRS seeks to link resources (planned and actual) to intended results for all
program expenditures and works to enhance decision-making by providing access
to accurate financial and non-financial performance information.
Departments
subject to the Policy are required to develop an MRRS, which is composed of the
following components, which are used as the basis for their reports to
Parliament:
- Strategic Outcomes (SO):
measurable long-term enduring benefits to Canadians;
- Program Activity
Architecture (PAA): an inventory of programs linked to strategic outcomes,
including a supporting performance measurement framework (PMF), which sets
out the expected results to be achieved and the specific outputs to be
produced by a program along with performance measures; and
- Governance Structure: the
decision-making mechanisms, responsibilities and accountabilities within a
department.
The
performance measurement framework is a key component of the Policy on MRRS.
Organizations must identify performance measures for each strategic outcome and
program identified in their PAA.
The
Government recognizes that a strong performance measurement framework is an
important factor in facilitating evaluations. As required by the 2009 Policy
on Evaluation, heads of evaluation of all large departments and agencies
review and provide advice on departmental performance measurement frameworks
and strategies, and will also prepare annual reports for submission to their
departmental evaluation committees on the state of performance measurement
within their departments, in terms of how well it supports evaluation. These
reports will help departments identify needed improvements in performance
measurement to support the conduct of evaluations.
The
development of performance information is an ongoing process and significant
progress has been made in all three departments since the audit was conducted.
The three departments continue to be engaged and to work with the Treasury
Board of Canada Secretariat to improve their performance measurement frameworks
and related performance measurement information in order to better support
management and decision making.
As highlighted below, Environment Canada, Fisheries and
Oceans Canada and Human Resources and Skills Development Canada have all made
modifications since the audit to further improve and strengthen the quality of
their PMFs.
Fisheries
and Oceans Canada
Fisheries and Oceans
Canada has undertaken a complete redesign of its Management, Resources and
Results Structure (MRRS) for 2011-12, of which a key component is its
Performance Measurement Framework (PMF).
Fisheries and Oceans
Canada has made considerable efforts to increase the level of precision and
quality of performance indicators so that it provides a more effective and
reliable means by which performance may be assessed and reported to Canadians.
Further, Fisheries and
Oceans’ performance indicators for 2011-12 have been developed with input from
the departmental Evaluation Directorate to ensure they will support upcoming
evaluations, and will be subjected to periodic internal and external reviews to
ensure continuous improvement.
Fisheries and Oceans
Canada has also launched a Performance Measurement Action Plan, which is a
comprehensive phased approach to ensure that Performance Measurement Strategies
are developed and implemented for all departmental programs by December 2011.
These Performance Measurement Strategies will support both the Performance
Measurement Framework of the department as well as individual evaluations.
Environment
Canada
Since the OAG audit
was conducted, Environment Canada undertook a major revision to its
Program Activity Architecture (PAA) and Performance Measurement Framework
(PMF). The PMF in effect for the 2010-11 fiscal year includes coverage of all
PAA programs.
Performance
measurement strategies are being renewed or developed for all PAA programs
over several years and on a phased basis, with the timing of these
strategies aligned to the schedule of program evaluations.
An inventory of
performance indicators currently in use department-wide over and above those in
the PMF is being developed; these will provide a baseline against which
proposed changes to indicators may be assessed, and this work will enable the
identification of important gaps that need to be addressed.
As programs proceed to
implement systems to collect and analyze performance data, areas where
revisions are needed to ensure relevance, reliability and utility of reported
information are identified through a process of regular review and challenge.
Environment Canada’s Audit and Evaluation Branch will review the implementation of the PMF and
program performance measurement strategies through several mechanisms. For
example, pre-evaluation evaluability assessments will be conducted two years
prior to each planned evaluation and will identify possible issues with the
availability or quality of performance information as well as other issues
related to evaluability. Implementation of management responses to
recommendations in evaluations, particularly those dealing with performance
measurement, will continue to be monitored regularly.
Human Resources Skills
Development Canada
In 2009, HRSDC
undertook a broad-based review of its Departmental Performance Measurement
Framework (PMF) using Assistant Deputy Minister Strategic Outcome Planning
Tables as the forum for discussion. This resulted in a strengthened 2010-11 PMF
with improved expected results statements and performance indicators.
As part of its annual
MRRS update, HRSDC will be looking to further improve specific indicators
within the PMF for 2011-12, following a review by senior management and the
Departmental Audit Committee. As with previous years, the Evaluation
Directorate provided input to the development of the 2011-12 departmental
performance indicators. In preparation for 2012-13, the Department will undergo
a broader review of the departmental PMF.
HRSDC supports
evidence-based policy and program development, diagnostic exercises, forecasting
and evaluation, as well as the collection and analysis of data in support of
improving indicators and data reliability. In support of this function and to
strengthen management of PMF-related information, HRSDC is currently developing
a web-based application for storing and sharing performance measurement
information and data. This central repository of current and historical
information on the PMF will enable both increased reporting flexibility and
comparability of performance data across several years.
Recommendation 3 That the Treasury Board of Canada
Secretariat include in the annual report on the status of the evaluation
function, an update on its oversight and support activities for effectiveness
evaluation.
Recommendation 5 That the Treasury Board of Canada
Secretariat include in the annual report on the status of the evaluation
function, information on the number of effectiveness evaluations, and in which
departments, that had difficulty assessing program success due to insufficient
performance information.
Recommendation 6 That the Treasury Board of Canada
Secretariat include in its annual report on the status of the evaluation
function, information on the percentage of direct program spending and grants
and contributions programs covered by evaluations in the fiscal year, as well
as the percentage evaluated over the five year cycle.
Recommendation 7 That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function,
information on the number of evaluators employed in large departments in the
federal government.
Recommendation 8 That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function
information on the percentage of evaluations that are conducted by contractors.
The
recommendations are accepted, and a clarification on roles and responsibilities
is provided.
As defined under
the 2009 Policy on Evaluation, evaluations are required to assess the
value for money of programs, including the extent of program effectiveness,
efficiency, economy and relevance. As indicated in the Government Response to
recommendation 2, TBS will prepare an annual Health of the Evaluation Function
Report, as required under the 2009 Policy on Evaluation. The report will
contain information about the status of the evaluation function, its progress
in meeting legislated requirements for evaluation of ongoing programs of grants
and contributions under the Financial Administration Act, as well as its
progress in implementing the 2009 Policy on Evaluation. The Health of
the Evaluation Function Report will also contain information identified in
PAC’s recommendations, including information on:
- TBS’ oversight and support
activities for the evaluation function;
- the number of evaluations
TBS has received from departments;
- the percentage of direct
program spending and grants and contribution programs covered by
evaluations in the reporting year, as well as the percentage evaluated
over the five-year cycle;
- the number of evaluators
employed in large departments in the federal government;
- the use of contractors in
the conduct of evaluations; and
- the percentage of large
departments assessed under the Management Accountability Framework whose
evaluations cited difficulties in assessing program effectiveness due to
insufficient performance information.
It is the
responsibility of individual departments to report instances where their
evaluations had difficulty assessing program success due to insufficient
performance information, as under the Standard on Evaluation for the
Government of Canada, every departmental evaluation report must disclose
“the constraints of the evaluation with respect to the design, methods and
consultations.” Further, heads of evaluation in large departments and agencies
are required by the Directive on the Evaluation Function to submit to
their Departmental Evaluation Committee an annual report on the “state of
performance measurement of programs in support of evaluation”.
Recommendation 9
The Office of the Auditor General conduct a
follow-up audit within five years of the initial audit on whether departments
are able to meet the requirements for effectiveness evaluation, and whether
they are conducting evaluations in an economical and efficient manner.
The
Government Response does not address the recommendations that fall outside its
area of responsibility. As such, the Government is unable to respond to the
Committee’s recommendation 9, which is directed for consideration of the Office
of the Auditor General. The Auditor General of Canada has authority to respond
to this recommendation independently in her capacity as an agent of Parliament
DEPARTMENT OF CANADIAN HERITAGE
Action Plan for the
Implementation of the Auditor General’s Recommendations
on “Evaluating the Effectiveness
of Programs”
RECOMMENDATION |
DEPARTMENTAL
RESPONSE |
ACTION PLAN / STATUS |
1.37 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment
Canada, Fisheries and Oceans Canada, and Human Resources and Skills
Development Canada should develop and implement action plans to ensure that
ongoing program performance information is collected to support effectiveness
evaluation. |
AGREED
1.
The Department is executing its Action Plan for Implementation of the
Management, Resources and Results Structures (MRRS) Policy to ensure that
program staff are able to fulfill their responsibility for developing and
maintaining performance measurement strategies. Action plan measures include:
the provision of information
sessions and workshops,
the establishment of indicators
and corresponding targets,
the development of robust
methodologies to demonstrate outcomes,
the establishment of a centre
of expertise on performance measurement within the Department,
the design and implementation
of adequate tools and guidelines,
the establishment of relevant
information technology and systems, and
the regular analysis and
reporting of collected data.
This
action plan is expected to be completed by end of the 2011–12 fiscal year. |
PARTIALLY
IMPLEMENTED
IN
PROGRESS –
Adjustments
were made to the MRRS Implementation Action Plan due to departmental budget
reductions and the launch of the Transformation Initiative. The Action Plan
will be updated once the Transformation Initiative and the implementation of
the Internal Services Review, in particular, are concluded.
In
the meantime, the department continues to make progress in improving its
state of performance measurement.
Provision
of Information Sessions and Workshops: In August 2009, the Deputy
Minister of PCH identified performance measurement as a learning priority. In
September 2009, the Strategic Planning Directorate (SPD) provided performance
measurement information and workshop sessions to all Program Managers,
Program Directors and Program Officers. Human Resources Intranet has listed
training on Performance Measurement as a learning priority for 2010-2012 for
all PCH employees. SPD is therefore engaged in delivering two performance
measurement training sessions in November, 2010. More in-depth training may
be offered during Spring 2011. In the meantime, SPD is working with the
Centre of Expertise (CoE) on |
RECOMMENDATION |
DEPARTMENTAL
RESPONSE |
ACTION PLAN / STATUS |
| |
|
Grants and Contributions,
Evaluation Services Directorate (ESD) and Financial Management Branch (FMB)
towards the adoption of a common terminology with regards to performance
management throughout the Department. SPD is also giving advice to branches
and programs on performance measurement strategies and their alignment with
the Performance Measurement Framework (PMF).
Establishment of Indicators and
Corresponding Targets: PCH continues to improve its Performance Measurement Framework. The PMF has
continued to mature and evolve. The 2011-2012 PMF contains improved
indicators and an increased number of targets. Pending Transformation
Initiative organizational changes, the revised action plan will outline next
steps and scope of modifications to the PMF for the coming months.
Development of Robust
Methodologies to Demonstrate Outcomes: PCH undertook a PMF
methodology development exercise in summer 2009 that resulted in the
refinement of indicators and expected results at SO and PA levels. In
addition, targets have been identified for most of the PA and PSA.
Establishment
of a Centre of Expertise on Performance Measurement within the Department: PCH has (staffed) a
Manager and an analyst to the performance measurement file. The establishment
of a Centre of Expertise is on hold due to current fiscal restraints and
pending further decisions on delivery models of corporate functions. Those
decisions are expected to be implemented in the beginning of 2011.
Additionally, the SPD has implemented a formal planning network, notably
through the development of a |
RECOMMENDATION |
DEPARTMENTAL
RESPONSE |
ACTION PLAN / STATUS |
| |
|
Level IV (Director- and
Manager-level) Planning Working Group, which supports the Level III
Integrated Planning Committee in the governance structure.
Design and Implementation of
Adequate Tools and Guidelines: PCH has developed methodology fiches to elaborate
on expected results and indicators at the SO and PA level. These were very
helpful and used for reporting requirements like the 2009-2010 DPR. Further
development of the sub-sub-activity level is discretionary and has been
delayed due to other priorities; however, it will be revisited at a later
date.
Establishment of Relevant
Information Technology and Systems: This project was cancelled due
to fiscal restraints. Further direction will be given once the results of the
Transformation Initiative have been implemented. Details will then unfold.
Regular Analysis and Reporting
of Collected Data: PCH has been piloting the data collection process within
the Heritage Group (Program Activity 3) for two fiscal years, using
2008-2009 as a reference year. The 2009-2010 data collection contributed to
the improvement of the reporting capacity.
PCH will prepare to expand the
data collection, analysis and reporting during FY2011-2012. |
| |
2.
The Department’s Office of the Chief Audit and Evaluation Executive is
continually providing advice and support to Department managers in their
efforts to implement this action plan. In line with the Transfer Payment Policy,
this office is also providing timely advice and support on program design and
performance measurement strategies |
COMPLETED – PCH’s OCAEE will
continue to systematically review and provide sign-off on all TB Submissions,
ensuring the quality and validity of information provided on evaluations and
performance measurement strategies. Evaluation managers will also continue to
review logic models, performance indicators and evaluation strategies for
programs in their portfolio. This structure ensures coverage of the entire
departmental |
RECOMMENDATION |
DEPARTMENTAL
RESPONSE |
ACTION PLAN / STATUS |
| |
through
the review of official approval documents for the creation and renewal of new
or existing programs. |
Program
Activity Architecture and enables better monitoring of the quality of
performance measurement strategies throughout the department. |
| |
3.
Finally, as required under the new Evaluation Policy, the office will be
submitting, in the 2010–11 fiscal year, its first annual report to the
departmental evaluation committee on the state of performance measurement in
the Department. |
PARTIALLY
IMPLEMENTED
Phase
I – Preliminary Study
The
Chief Audit and Evaluation Executive completed a preliminary assessment on
the state of performance measurement of programs at PCH in support of
evaluation. The main objective was to assess the appropriateness and
availability of data collected by programs for effectiveness evaluations.
Broad results were presented to the Departmental Evaluation Committee late
June 2010.
Phase
II – Study
This
phase corresponds to the actual conduct of the assessment. Some of the issues
to be addressed include the following:
portrait of results data
availability and collection methods;
actions taken by program
management and corporate functions to implement and improve performance
measurement;
factors affecting the
implementation of a performance culture at PCH, including support offered by
PCH senior management;
training offered to program
staff and training required;
clarity of the roles and
responsibilities in the Department for ensuring the adequate implementation
of performance measurement strategies.
Deliverable: A final report
including:
assessment of the state of
performance measurement at PCH in support to evaluation;
recommendations and suggested
methodologies to senior management and programs to improve and strengthen
performance measurement within PCH.
Timeline: March 2011. |
RECOMMENDATION |
DEPARTMENTAL
RESPONSE |
ACTION PLAN / STATUS |
1.43 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment
Canada, Fisheries and Oceans Canada, and Human Resources and Skills
Development Canada should consider the merits of including external experts
on their departmental evaluation committees. The Treasury Board of Canada
Secretariat should provide guidance to departments in this regard. |
AGREED
At
the Department, oversight of the evaluation function is provided by the
Strategic Policy, Planning and Evaluation Committee (SPPEC), chaired by the
Deputy Head. This structure provides opportunities for enhanced integration
between the policy, planning, and evaluation functions of the department. As
well, the Department already brings key evaluation reports when necessary to
its department audit committee for review and discussion. The Department will
collaborate with the Treasury Board of Canada Secretariat’s Centre of
Excellence in Evaluation to assess the value added of integrating the advice
of external evaluation experts to inform the work of departmental evaluation
committees. |
COMPLETED
1.
The Department’s Chief Audit and Evaluation Executive will continue, in
future, to bring all key evaluation reports to its departmental audit
committee for review and discussion.
2.
PCH already collaborates with Treasury Board of Canada Secretariat’s Centre
of Excellence for Evaluation (CEE) on several evaluation related issues. The
Department is well represented and actively contributes to all
interdepartmental working groups put in place by the CEE for the
implementation of the 2009 Policy on Evaluation. PCH participated to
CEE’s consultation and discussions with departments to clarify the role of
external evaluation experts within departmental evaluation committees. PCH
provided the CEE with comments on the challenges related to the
implementation of this recommendation at PCH, comments that helped develop
guidance on this issue. |
1.48 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Fisheries and
Oceans Canada, and Human Resources and Skills Development Canada should
implement systematic processes to determine whether their effectiveness
evaluations are meeting government-wide requirements and internal corporate
needs, and act on areas identified for improvement. |
AGREED
The
Department recognizes the need to establish systematic processes to assess
whether effectiveness evaluations are addressing needs. The Evaluation
Services Directorate is already developing a performance measurement
framework and management strategy to identify clear performance expectations
and standards for the Department’s evaluation function. A systematic process
to collect, analyze, and report on performance data and client satisfaction
will be implemented in order to identify |
PARTIALLY
IMPLEMENTED
Meeting
government-wide requirements:
COMPLETED – 1.The first draft
of the final report for each evaluation conducted by PCH Evaluation Services
Directorate is peer-review internally. A standardized tool is used
systematically to assess different criteria and capture relevant comments in
accordance with the Standards of Evaluation for the Government of Canada and
Treasury Board Secretariat’s MAF criteria for the quality of evaluations. |
RECOMMENDATION |
DEPARTMENTAL
RESPONSE |
ACTION PLAN / STATUS |
The
Treasury Board of Canada Secretariat should monitor and provide any
additional support it considers necessary for the implementation of these
processes. |
areas
of improvements. Based on data collected during the first year of
implementation (the 2010–11 fiscal year), periodic reporting to the
Departmental Evaluation Committee should begin by the 2011–12 fiscal year.
This performance information will complement data already reported in the
context of the annual management accountability framework assessments
conducted by the Treasury Board of Canada Secretariat. |
Meeting
internal corporate needs and act on areas identified for improvement:
COMPLETED
– 1. Program managers, DGs, ADMs and the Deputy Minister are always consulted
on the 5-year Departmental Evaluation Plan to ensure that programming
adequately address specific reporting needs. Terms of reference and
evaluation questions for high risk programs are discussed and approved by the
departmental evaluation committee chaired by the Deputy Minister.
IN
PROGRESS – 2. The Evaluation Services Directorate has reviewed its own logic
model and developed a performance measurement framework to monitor and assess
progress towards achieving the expected results. Clear performance
expectations have been identified and are part of the Department’s Office of
the Chief Audit and Evaluation Executive 2010-11 / 2012-13 Integrated
Business Plan. Fiscal year 2010-2011 will be dedicated to develop a baseline
of data, 2011-2012 to implement the data collection system and by April 1
2012, the Evaluation Services Directorate will review and update its service
charter and inform its departmental clients. Once implemented, service
delivery will be monitored to assess compliance and make necessary
adjustments.
IN
PROGRESS – 3.
The Evaluation Services
Directorate developed and piloted a client survey questionnaire to collect
client satisfaction data after the conduct of an evaluation. The
questionnaire was updated and programmed to be accessed online. Effective
January 2011, this tool will systematically be used to collect performance
data and identify areas of improvement.
Also,
effective in January 2011, the Evaluation Services Directorate will
systematically hold and document |
RECOMMENDATION |
DEPARTMENTAL
RESPONSE |
ACTION PLAN / STATUS |
| |
|
lessons learned sessions with
program management after the completion of an evaluation. The goal of these
sessions will be to review the evaluation process, assess how well program
management’s needs were met, highlight best practices and identify possible
improvements.
The Evaluation Services
Directorate is developing an annual client survey to collect additional data
on the quality and timeliness of consulting services/advices, as well as
usage of evaluation results. This tool will be implemented in the fiscal year
2011-2012. |
Department of
Human Resources and Skills Development Canada
Action Plan for the
Implementation of the Auditor General’s Recommendations
on “Evaluating the Effectiveness
of Programs”
AUDITOR GENERAL OF CANADA RECOMMENDATIONS |
MANAGEMENT
RESPONSE |
MANAGEMENT
PLAN
ACTION(S)
TO BE UNDERTAKEN |
|
|
RECOMMENDATION
1.37 Agriculture
and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada,
Environment Canada, Fisheries and Oceans Canada, and Human Resources and
Skills Development Canada should develop and implement action plans to ensure
that ongoing program performance information is collected to support
effectiveness evaluation. |
Human
Resources and Skills Development Canada’s response.
Agreed.
The Department accepts this recommendation that echoes the intent of various
Treasury Board of Canada Secretariat policies, including the 2009 Evaluation
Policy. As such, the Department already gathers and monitors ongoing
performance information to support effectiveness evaluation, including:
• monitoring implementation of
management responses to previous evaluations that have identified data issues
for current programs;
• undertaking early evaluative
work in advance of the formal initiation of effectiveness evaluations as part
of the evaluation planning process, to review the state of performance data
and logic models; and |
Develop
and submit an annual report on the state of performance measurement, as
required by the Policy on Evaluation.
Management
response follow-up study (2004-2005 through 2006-2007) was submitted to the
Departmental Evaluation Committee (DEC) on July 7, 2009. Frequency and
approach to future monitoring to be discussed at DEC.
Ongoing |
March 2010
(annual updates)
December 8, 2009
(periodic reporting)
Ongoing |
DG Evaluation,
Strategic Policy and Research
Branch
DG Evaluation,
Strategic Policy and Research
Branch
DG Evaluation,
Strategic Policy and Research
Branch |
AUDITOR GENERAL OF CANADA RECOMMENDATIONS |
MANAGEMENT
RESPONSE |
MANAGEMENT
PLAN
ACTION(S)
TO BE UNDERTAKEN |
|
|
| |
• monitoring the implementation
of new programs (for example, Economic Action Plan initiatives) to ensure the
necessary administrative and performance data are available to support future
evaluation activities.
The
Department is also undertaking a comprehensive review, refinement, and
validation of its Performance Measurement Framework (PMF) to make it more
robust and comprehensive, to support ongoing planning, monitoring, and
managing for results and the Evaluation Directorate has been actively
involved in this work. The progress made on the departmental PMF will support
both performance monitoring as well as evaluation of relevance and
effectiveness. |
Ongoing
Complete
a comprehensive review, refinement, and validation of the Performance
Measurement Framework (PMF) with engagement by ADMs across the Department.
Work
with Treasury Board Secretariat to continue to strengthen PMF on an ongoing
basis. |
Ongoing
September,
2009 (complete)
Annual
updates |
DG Evaluation,
Strategic Policy and Research
Branch
DG Corporate Planning and
Accountability, Strategic Policy and Research Branch |
AUDITOR GENERAL OF CANADA RECOMMENDATIONS |
MANAGEMENT
RESPONSE |
MANAGEMENT
PLAN
ACTION(S)
TO BE UNDERTAKEN |
|
|
RECOMMENDATION. 1.43 Agriculture and Agri-food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment
Canada, Fisheries and Oceans Canada, and Human Resources and Skills
Development Canada should consider the merits
of including external experts on their departmental evaluation
committees. The Treasury Board of Canada Secretariat should provide guidance
to departments in this regard. |
Human
Resources and Skills Development Canada’s response.
Agreed.
The Department has, in the past, included members from outside the Department
on its departmental evaluation committee. The Department will reconsider the
formal inclusion of external experts on the current Departmental Evaluation
Committee and will look to the Treasury Board of Canada Secretariat for
guidance on this part of the Evaluation Policy. |
Discuss
external membership at the Departmental Evaluation Committee, and revise
terms of reference as required. |
November 17, 2009 |
DG Evaluation,
Strategic Policy and Research
Branch |
AUDITOR GENERAL OF CANADA RECOMMENDATIONS |
MANAGEMENT
RESPONSE |
MANAGEMENT
PLAN
ACTION(S)
TO BE UNDERTAKEN |
|
|
RECOMMENDATION 1.48
Agriculture and Agri-food Canada, Canadian Heritage, Citizenship
and Immigration Canada, Fisheries and Oceans Canada, and Human Resources and
Skills Development
Canada should implement systematic processes to determine whether
their effectiveness evaluations are meeting government-wide requirements and
internal corporate needs, and act on areas identified for improvement. The
Treasury Board of Canada Secretariat should monitor and provide any
additional support it considers necessary for the implementation of these
processes. |
Human
Resources and Skills Development Canada’s response.
Agreed.
The Department currently employs a variety of systematic processes to ensure
the quality and relevance of its evaluations for both internal and
government-wide needs. External peer reviewers and evaluation advisory
committees are a mandatory element of the evaluation work to ensure that
evaluations are meeting information needs. Further work in systematically
determining whether evaluations are meeting government and senior management
needs will be developed, building upon the current annual Management
Accountability Framework assessment process led by the Treasury Board of
Canada Secretariat. Areas for improvement will be identified and reported to
the Departmental Evaluation Committee. |
Develop
the Departmental Evaluation Plan, in consultation with senior management, to
identify the appropriate level, focus and sequencing of evaluation resources
as well as areas for improvement.
Identify possible areas of improvement through the annual
Management Accountability Framework assessment. Present and discuss at the
Departmental Evaluation Committee. |
April 2010
(annual updates)
April / May 2010
(annual process) |
DG Evaluation,
Strategic Policy and Research
Branch |
|