With a workforce of approximately 13,000 employees and an operating
budget of over $1.7 billion, the Canada Border Services Agency (CBSA)
manages the access of people and goods to Canada. In 2013–2014, the CBSA
administered close to 100 million travelers into Canada, cleared more than 14 million
commercial shipments, made more than 9,000 drug seizures, and removed almost
14,000 failed refugee claimants and other inadmissible persons. The CBSA also
collected $26.9 billion, which is approximately 10% of the Government of
Canada’s revenues.[1]
Like other departments and agencies, the CBSA relies upon
information technology (IT) to fulfil its mandated services. The CBSA’s
portfolio of IT investments, managed by its Information, Science and Technology
Branch, is budgeted at more than $1 billion, primarily consisting of the Border
Modernization initiative, involving eight projects, and the Beyond the Border
Action Plan, which includes 32 government-wide initiatives.[2]
The Beyond the Border Action Plan aims to enhance border security
while expediting legitimate cross-border trade and travel. It seeks to “push
the border out,” by collecting information earlier and thereby facilitating the
flow of legitimate goods and travellers that are low risk, through such
programs as NEXUS, Trusted Trader and FAST.[3] The CBSA’s IT projects will help it realize operational benefits, such as
improving its lookout system, scrutinizing passenger name record information
before inbound flights depart, and analyzing electronic manifest information
before commercial import shipments arrive.[4]
In spring 2015, the Office of the Auditor General
of Canada (OAG) released a performance audit that examined whether the CBSA had
the corporate and management practices in place to enable the delivery of
IT investments that align with and support the CBSA’s strategic corporate
objectives.[5] The OAG examined five of the
CBSA’s IT projects:
- Entry/Exit Initiative;
- Field Operations Support System replacement project;
- Interactive Advance Passenger Information Initiative;
- Single Window Initiative; and
- Temporary Resident Biometrics Project.[6]
The House of Commons Standing Committee on Public Accounts (the
Committee) held a hearing on this audit on 27 May 2015.[7] From
the OAG, the Committee met with Michael Ferguson,
Auditor General of Canada, and Martin Dompierre,
principal. The CBSA was represented by Caroline Weber,
Vice-President, Corporate Affairs Branch; Louis-Paul
Normand, Associate Vice-President, Information, Science and Information
Technology Branch; and Chris Bucar, Acting Deputy Chief
Financial Officer and Director General Resource Management.
In response to an external review, the CBSA implemented, in 2013,
the Project Portfolio Management Framework to better manage its portfolio of IT
investments. The framework focuses on undertaking the right projects at the
right time and overseeing them to ensure that expected benefits are realized.[8] In order to assess the CBSA’s management practices, the OAG examined the CBSA’s
portfolio governance, investment planning and annual IT planning,
enterprise architecture, and the management of IT portfolio risk.[9]
The OAG found that the Transformation, Innovation and Project
Portfolio Committee, which is responsible for project portfolio oversight and
alignment, had not been given sufficiently detailed information to exercise its
responsibilities, and there was no ongoing assessment of how sub-portfolios
were strategically aligned and were within resource capacity.[10] While the CBSA has a
Project Management Framework, the OAG noted that projects were allowed to move
to subsequent phases of development even though prerequisite conditions under
the Framework, such as business cases, risk assessments, detailed project plans
and project benefit plans, were not met.[11]
Michael Ferguson, Auditor General of Canada, clarified the OAG’s
expectations with respect to applying the framework. He commented:
Going through projects and making sure that each
stage is met along the way is critically important to understand that all the
requirements as well as the time and the budget are on schedule, to make sure
the project is going to deliver what was intended. We would never ask a
department to go back and try to redo gates that have already been passed. But
we would expect the department, at the point they were putting in place a new
framework, to at least look at those projects that were in place and identify
if they seem to be on the right path to deliver what they were intended to
deliver, regardless of the fact that you can't go back and re-document certain
gates, whether they were passed or not passed.[12]
Caroline Weber, Vice-President of the Corporate Affairs Branch at
the CBSA, described how the CBSA was working to apply its framework to projects
that are in progress. She said:
The challenge for us has been IT projects that
were already in progress, had been started, were a long way down the road to
being completed when we bring in a new project management framework. The
question then was whether we were going to go back in order to check the boxes
on these projects, or whether we were going to continue and deliver, especially
when we're near the end of a project, knowing that we would leave it
undocumented, but thinking perhaps it wasn't the right way to use our resources
at that time. On a go-forward basis, all of our projects are being put through
the entire framework with no exceptions, and we review that all the time.[13]
The OAG observed that the CBSA’s investment plan, which outlines
investments in projects, assets and acquired services, had not been updated
since 2011, even though the Treasury Board Policy on Investment Planning—Assets
and Acquired Services requires the plan to be updated every three years, or
when a significant change occurs. The CBSA’s IT portfolio size has
significantly increased since then.[14] Nonetheless, the CBSA had conducted some ad hoc IT investment planning
activities. In April 2014, it developed a Five-year Capital Plan, in part to
manage the fact that capital funding surpluses had grown even though some
projects were delayed due to a lack of funding. In June 2014, the CBSA
completed an Annual IT Plan to demonstrate how IT aligns with the CBSA’s
strategic objectives, but the plan does not include IT projects. At the request
of the Treasury Board Secretariat of Canada, in August 2014, the CBSA prepared
an Integrated IT Project Plan that included 14 of the CBSA’s 30 projects.[15]
Ms. Weber told the Committee that the failure to update the
investment plan resulted “in some frequent and intensive conversations with our
colleagues in the Treasury Board Secretariat.”[16] However, Chris Bucar, Acting Director General of Resource Management at the
CBSA, informed the Committee that the CBSA’s “investment plan and annual IT
plan … were submitted to Treasury Board, and approved on April 23, 2015.”[17]
The OAG found that the CBSA does not have an overall portfolio
enterprise architecture, which can maximize the value of IT investments by
promoting reuse, agility and innovation.[18] According to the OAG, the lack of such an architecture resulted in the
development of duplicate IT systems. For example, two master data management
systems were built, with a future plan to merge them.[19] Additionally, the CBSA
ended its Business-to-Business project after completing phase 1, even though
several of its IT systems identified the need to develop business-to-business
services in order to communicate with the IT systems of other organizations.[20]
Ms. Weber indicated that the CBSA was working “to ensure that
enterprise architecture is adhered to by all IT projects through formal gate
reviews and approvals.”[21] She also said:
A functional directive covering every domain of
the enterprise architecture will be finalized by September, 2015. Also, we've
begun to move individual projects toward architecture standards that fully
align with Shared Services Canada's directions, and the service life-cycle
management framework ensures that enterprise architecture directions are
adhered to by all projects.[22]
While the CBSA conducted some risk assessments, the OAG found that
it did not have an overall risk profile for its IT investment portfolio.
Without this profile, the OAG concluded that the CBSA could not determine the
level of risk in proportion to the expected benefits of its portfolio of IT
projects.[23]
Ms. Weber told the Committee that the CBSA is “going to continue to
maintain [its] Beyond the Border project-level risk profile and to provide a
full portfolio risk update roll-up at the quarterly Beyond the Border senior
project advisory committee.”[24]
In order to encompass issues related to updating the investment
plan and annual IT plan, defining and completing target enterprise
architecture, and completing an IT project portfolio risk profile, the OAG
recommended that the CBSA ensure that all elements of the Project Portfolio
Management Framework are implemented.[25] When he released the audit in April, Mr. Ferguson commented, “We were very
happy with the framework that had been put in place in the agency and the fact
that it was comprehensive. Our concern, again, was that it wasn't at this point
in time always being applied in the management and the oversight of the
projects.”[26]
In response, Ms. Weber noted, “We did recognize a number of years
ago that we needed to improve with regard to IT project management, so we put
in place this framework and started implementing it. I can tell you that our IT
projects are being delivered on time now and on budget, so the framework that
we put in place has achieved that.”[27]
Clearly defining IT systems requirements allows organizations to
demonstrate that IT projects align with strategic directions, meet business
needs, and have appropriate funding and resources.[28] The OAG examined whether the
CBSA implemented IT project management practices and whether it demonstrated
results, in particular, business requirements and outcomes and benefits.[29]
The OAG noted that the projects examined did not have clear
corresponding IT systems requirements before the projects started the execution
phase, which led to several challenges:
- As noted above, there was a duplication of effort in the
development of master data management systems;
- As projects did not have costs broken down by deliverables, the
CBSA could not identify the costs of deferring work, leading to projects
requesting more funds than could be used in a given fiscal year;
- Differences of opinions with Citizenship and Immigration Canada
(CIC) over system functions for the replacement of the Field Operations Support
System (FOSS) led to project delays; and
- Some projects, such as the Interactive Advance Passenger
Information initiative, were at risk of not delivering what was expected.[30]
The Development of CBSA’s IT projects which meet necessary business
requirements is particularly challenging because of the CBSA’s dependence upon
external agencies in the ecosystem in which it works. The consequence is that
not all business requirements are specified up front and that policy change
decisions are sometimes made after a project begins.
Examples of external agencies upon which the CBSA is dependent in
developing one IT project alone (the Entry-Exit IT project) include:
- global partners;
- nine partners within the Government of Canada who request information;
- ninety acts of Parliament to comply with, including associated
regulations, with some still being awaited;
- privacy requirements; and
- the airline industry.[31]
With respect to the replacement for FOSS, Louis-Paul Normand,
Associate Vice-president of the Information, Science and Technology Branch at
the CBSA, informed the Committee that the decision for developing master data
management has not been made and was still under consideration.[32] In response to a question
about the cost of the delay in replacing FOSS, he said:
We had to agree as to what would be done in GCMS,
so in the CIC system, and what had to be done and developed by CBSA. It's not
an incremental cost, in that if we hadn't done it—in this case we decided to do
it—then CIC would have had to do it. The disagreement had to do with who was to
do it, not the cause of an overrun. The $2.3 million was the licence for the
FOSS vendor, the legacy vendor. The decision would have been to carry it on
anyway because of the rollout plan that we deployed. In other words, there was
no way that we would have been able to turn off FOSS as a back-up system in
December 2013 as originally planned.[33]
Of the five projects examined, the OAG found that all had defined
project outcomes at the planning phase, but four of the projects did not
define measureable benefits, which would have enabled the CBSA to demonstrate
alignment between project delivery and business needs.[34] Furthermore, in August
2014, the CBSA identified that over 50% of its portfolio projects had “low
readiness,” which means that there was minimal information on whether benefits
would be realized or aligned with strategic objectives, and the readiness of
another 27% of projects was unknown.[35]
The OAG recommended that the CBSA ensure that project requirements
are met and measures are defined to assess whether projects deliver expected
benefits.[36] Mr. Ferguson emphasized the importance of assessing benefits, stating, “There's
more than just making sure systems are delivered on time and on budget. There's
also making sure that the systems are delivering what they were intended to
deliver. That's why we talk in the report about the need to assess the benefits
and assess whether the systems are aligned with the objective of the agency.”[37]
Mr. Normand described the CBSA’s response, commenting, “We've
mandated that all active projects go back and revisit their business case,
their business requirements, and their project plan to make sure that the
benefits are clearly identified, measurable, and harvestable.”[38]
Project dashboards, which show the status of a project or portfolio
of projects, can be used by senior management to make decisions on whether IT
projects require more active monitoring and whether corrective action is
needed.[39] The CBSA uses project dashboards that show the health of individual projects,
i.e., whether they are meeting schedule milestones, undergoing changes in scope
and staying within budget, as well as the enterprise dashboard, which is a
consolidation of project dashboards and is provided to the executive committee.[40] The OAG examined whether
the information reported to decision makers on IT projects was in accordance
with the CBSA’s frameworks and guidelines.[41]
The OAG found that the information provided in project dashboards
was inconsistent and incomplete.[42] For example, there were variances that ranged between 6% and 157% with regards
to the financial information reported on project dashboards when compared to
other CBSA reports.[43] Additionally, project managers had discretion in what they reported on the
dashboards, which influenced a project’s health status.[44] The OAG also observed
variances in the reporting of project status and project scope between project
dashboards and other documentation.[45]
The OAG noted that in May 2014, the CBSA introduced a new process
to monitor project performance using value management reporting to mitigate
discrepancies in the monthly project dashboards.[46] This method can improve
project performance monitoring, but the OAG noted some inconsistencies between
value management reports and project dashboards.[47]
Mr. Normand described the benefits of value management reporting at
the CBSA, observing:
The first requirement is that you need to have a
good plan. Based on that plan, you load up with your resources that you need to
deliver on that plan. Then the tracking becomes the planned value. As you go
through six months of activities, you should be 50% done on a 12-month project.
If your actuals, your financials, are over that or below that, it creates a
variance that you track. That's the essence of earned value. It does that for
scheduling variance. It does that for cost variance. Now in our dashboard to Treasury
Board, the colour for cost and schedule is no longer subjective. It is based on
a percentage of variance on cost or scheduled variance.[48]
As the absence of consistent controls over project reporting can
raise risks that reporting on project status is ineffective, the OAG
recommended that the CBSA establish clear procedures and practices on how the
information for project dashboards is collected, reported and enforced.[49]
Ms. Weber acknowledged the issues raised by the OAG, stating:
We did have some discrepancies, as the Auditor
General referred to. We knew about those discrepancies, but we were living with
them because there were time gaps or time lags between some of the information.
We were not including some standard information that would have been easy to
add because we knew it was there and it's not the way we usually reflect things
internally. However, we are changing that as a result of this audit.[50]
She added that in response, the CBSA is “developing a baseline set
of performance benefits indicators and quarterly reporting to the executive
cadre on benefits realization status of IT projects; and initiating a formal
review process of the procedures and practices of how project dashboard
information is collected, reported, and enforced.”[51]
The CBSA provided an action plan to the Committee which outlined a
number of commitments to improve the management of its portfolio of IT
projects. As the latest completion date listed in the action plan is March
2016, the Committee recommends:
RECOMMENDATION
That, by 31 March 2016, the Canada Border Services Agency provide
the Standing Committee on Public Accounts with a report outlining its progress
in addressing the Office of the Auditor General of Canada’s recommendations
contained in Report 5 of the Spring 2015 Reports.
The CBSA has a large portfolio of IT projects, many of which are
essential to the effective delivery of its strategic objectives, in addition to
improving the efficiency of its operations. For example, the Single Window
Initiative allows data for several different federal departments to be
collected in a single transaction. In order to manage these complex projects,
decision-makers at the CBSA need to have access to complete and reliable
project information, with clear business requirements.
Overall, the OAG concluded that the CBSA had the necessary
corporate and management practices to deliver on its IT investments. However,
the CBSA had not put into practice all of the elements of its project portfolio
management framework to ensure that its IT projects would support strategic
corporate objectives. Also, the committees overseeing project portfolios did
not have sufficient information to fully exercise their responsibility to
ensure that project requirements and conditions were met at each stage of
approval.[52]
The CBSA has recognized that it needs to improve the implementation
of its project management framework. The Committee expects that the CBSA will
strengthen the governance process for IT investments, clarify IT systems
requirements to ensure project benefits are realized, and establish consistent
procedures and practices for the collection and reporting of project status
information.
|