Evaluation of Computers for Schools

PDF version

631 KB, 29 pages

March 2017

Presented to the Performance Measurement and Evaluation Committee on April 20, 2017

Approved by the Deputy Minister on May 29, 2017

Table of contents

• Executive summary
• 1.0 Introduction
  • 1.1 Report overview
  • 1.2 Program description
  • 1.3 Logic model
• 2.0 Methodology
  • 2.1 Evaluation objective and scope
  • 2.2 Evaluation issues and questions
  • 2.3 Data collection methods
  • 2.4 Limitations
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions and recommendation
  • 4.1 Relevance
  • 4.2 Performance
  • 4.3 Recommendation
• Management Response and Action Plan
• Summary of the evaluation of the Computers for Schools Program

List of acronyms

AEB

Audit and Evaluation Branch

CA

Contribution Agreement

CAP

Community Access Program

CFS

Computers for Schools

CFSC

Computers for Success Canada

EPSC

Electronics Product Stewardship Canada

ESDC

Employment and Social Development Canada

e-waste

Electronic waste

Gs&Cs

Grants and Contributions

ICT

Information and Communications Technology

ISED

Innovation, Science and Economic Development Canada

IT

Information Technology

ITU

International Telecommunication Union

OECD

Organization for Economic Cooperation and Development

O&M

Operations and Maintenance

RPP

Report on Plans and Priorities

SITT

Spectrum, Information Technologies and Telecommunications

TWEP

Technical Work Experience Program

YES

Youth Employment Strategy

List of tables

• Table 1: Partner contributions as a percentage of eligible recipients funding
• Table 2: Units donated and sent to recycling

List of figures

• Figure 1: CFS Logic model
• Figure 2: Distribution of computers by type of organization
• Figure 3: Total salary and O&M expenditures by year, CFS and TWEP combined

Executive summary

Program overview

Launched in 1993, Computers for Schools (CFS) is a national program that makes use of surplus computers from federal departments, provincial/territorial governments and the private sector. Computers are donated to CFS refurbishment centres, where they are refurbished for use by the program's beneficiaries, which include schools, libraries, registered not-for-profit organizations and Aboriginal communities.

CFS recipients may also offer youth experience refurbishing computers through the Technical Work Experience Program (TWEP), funded by the Government of Canada's Youth Employment Strategy (YES). Youth are engaged in the refurbishment process, and develop skills in computer repair and software testing while cultivating softer skills.

Between April 1, 2011 and March 31, 2016, CFS expenditures totaled $42.1 million, of which $3.3 million was for operating expenditures and $38.8 million^{Footnote 1} for Grants and Contributions (Gs&Cs).

Evaluation purpose and methodology

In accordance with the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of CFS. The evaluation covered the period from 2011–12 to 2015–16 Findings and conclusions are based on the analysis of multiple lines of evidence. The methodology included a document review, literature review, interviews, survey of current and former youth participants, site visits and an analysis of performance and financial data.

Findings

Relevance

There is a continued need to provide refurbished computers to students and other Canadians CFS provides an environmentally responsible means for governments and businesses to dispose of surplus computers. Additionally, providing youth with internships gives them opportunities to develop the skills necessary to enter the workforce.

CFS aligns with federal responsibilities to foster access to technology, maximize the use of crown assets and contribute to sustainable development through the appropriate disposal of information technology (IT) equipment. The CFS program does not duplicate or overlap any other government program. Further, the objectives of the program are consistent with federal government priorities related to developing stronger digital skills among Canadians, providing work experience to youth and reducing the Government's environmental footprint.

Performance

CFS has distributed nearly 370,000 refurbished computers to beneficiaries over the past five years, with the majority delivered to schools. In addition to providing technology to schools and assisting not-for-profit organizations, the recent program expansion gives access to computer technology to Canadians who could not otherwise afford it. Further, the program reduces the environmental footprint of government and businesses through reuse and recycling of their computer equipment.

With respect to work experience, CFS enhances the employability of youth by providing hands-on experience in the IT field and assists in the development of both Information and Communications Technology (ICT) and soft skills. About 300 interns were employed annually in CFS workshops across Canada, exceeding annual published targets The program has contributed to learning opportunities for youth. There is an opportunity for the program to recruit additional female participants.

The network of partners surrounding CFS brings with it substantial in-kind and cash donations. These donations enable the program to operate and succeed.

The program demonstrates economy and efficiency and continues to meet delivery targets despite a reduction in resources over the assessment period. However, the program continues to mine data manually, resulting in data quality issues and some challenges with program reporting.

Recommendation

The findings of the evaluation led to the following recommendation:

1. The CFS program should consider modernizing its data collection, capture and storage with a view to ensuring adequate performance information is available.
2. The CFS program should continue to explore the diversity of its interns and consider what more could be done to attract female candidates.

1.0 Introduction

1.1 Report Overview

This report presents the results of an evaluation of CFS.

The purpose of the evaluation was to assess the relevance and performance of CFS. The report is organized in four sections:

• Section 1 provides the program context and profile of CFS;
• Section 2 presents the evaluation methodology along with a discussion of data limitations;
• Section 3 presents the findings pertaining to the evaluation issues of relevance and performance; and
• Section 4 summarizes the study's conclusions and provides recommendations

1.2 Program description

Launched in 1993, CFS is a national program that makes use of surplus computers from federal departments, provincial/territorial governments and the private sector Computers are donated to CFS refurbishment centres, where they are refurbished. They are then distributed to the program's beneficiaries, including schools, libraries, registered not-for-profit organizations and Aboriginal communities.

Across the country, there are 13 organizations – called eligible recipients – operating almost 30 refurbishment centres. Operational models vary among eligible recipients depending on the resources available/donated. For example, one recipient may have to pay rent and use more volunteers, while another may have secured donated space and be able to retain more paid staff.

Eligible recipients offer students and recent graduates professional experience refurbishing computers through internships. Participating students develop skills in computer repair and software testing while cultivating softer skills such as time management and team building. The Technical Work Experience Program, funded by the Government of Canada's Youth Employment Strategy (YES)^{Footnote 2}, is a key enabler of CFS as it provides CFS with the majority of its refurbishment labour force.

Each year, Innovation, Science and Economic Development (ISED) consults with eligible recipients to determine the minimum standard computer that will be available free of charge to all beneficiaries requesting equipment. There is no minimum standard for donations, as all donated computers can be used for spare parts if they cannot be refurbished, with the remainder sold to recyclers. Beneficiaries may request upgrades to the standard, such as additional memory or a new hard drive, and in these cases, the upgrades are generally provided for a fee.

Budget 2015 announced an increased investment of $2 million over two years to expand the CFS program, extending access to refurbished computer equipment to non-profit organizations such as those that support low-income Canadians, seniors and new Canadians. The program Terms and Conditions were subsequently amended^{Footnote 3} to include these groups and to permit individuals to be ultimate program beneficiaries.

1.2.1 Delivery and Governance of the Program

The program is delivered through not-for-profit funding recipients (called eligible recipients) in each province and territory, with the exception of Nunavut which is partially being served by the recipient in the Northwest Territories. There is an additional recipient located in Gatineau, which handles the refurbishment of computers from the National Capital Region and equipment redistribution to recipients across the country. Finally, there is Computers for Success Canada (CFSC), an organization which plays a national coordinating role for CFS in partnership-building, communications and on specific initiatives such as #WelcomeRefugees.

Eligible recipients prepare and submit funding proposals to ISED Contribution Agreements (CAs) are prepared and signed, specifying the responsibilities of each party, the items for which expenditures are anticipated and eligible for reimbursement, the conditions under which payments are made, and mutually-agreed-upon targets designed to assess the success of the program in attaining its objectives Agreements comply with the requirements of the Treasury Board Policy on Transfer Payments and the Financial Administration Act and are reflective of Government of Canada policy and guidelines relating to Materiel Management and Disposal of Surplus Materiel.

Additionally, the Treasury Board Directive on the Disposal of Surplus Materiel states: (4.1) Departments must offer right of first refusal of all surplus personal computers, laptops, servers, monitors, keyboards, mice, printers, modems, hubs, network cards, hard drives, etc., to the Computers for Schools Program.

The program office ensures that applicable Government of Canada policies, procedures and practices are appropriately followed through the implementation of CAs, licensing agreements, site visits and regular contact with recipient organizations. Various federal departments provide advice or assistance to the program on the development of its internal policies and procedures:

• Communications Security Establishment Canada provides advice on security aspects and supports the approval of the disc-wiping software;
• Treasury Board upholds and maintains a Materiels Management community that supports and reinforces the "first right of refusal" granted to CFS;
• Public Services and Procurement Canada oversees the greening of government operations in collaboration with other federal departments by providing advice and guidance, as well as managing the Standing Offer for the recycling of federal assets; and
• Environment and Climate Change Canada provides advice on the Federal Sustainable Development Strategy in relation to air, water and nature, and on reducing the toxic and chemical impacts on the environment.

In addition, ISED works with Employment and Social Development Canada (ESDC) to coordinate the youth employment component of the CFS program.

CFS is delivered and monitored by the Spectrum, Information Technologies and Telecommunications (SITT) Sector of ISED.

1.2.2 Program Resources and Expenditures

Between April 1, 2011 and March 31, 2016, CFS expenditures totaled $42.1 million, of which $3.3 million was for operating expenditures and $38.8 million was for Grants and Contributions.

1.3 Logic model

The logic model depicts CFS program theory It shows how the program's activities are expected to lead to certain outputs and various levels of outcomes. The program's logic model was updated through consultation with program staff prior to the evaluation. The current CFS logic model is presented in Figure 1.

Figure 1: CFS Logic model

2.0 Methodology

2.1 Evaluation objective and scope

An evaluation of CFS is required under section 42.1 of the Financial Administration Act The program was previously evaluated in 2011–12.

In accordance with the Treasury Board Directive on the Evaluation Function, the purpose of this evaluation was to assess the core evaluation issues of relevance and performance The evaluation covered the five-year period of 2011–12 to 2015–16.

2.2 Evaluation issues and questions

Based on the program's Performance Measurement Strategy, and subsequent consultations with the program area, the evaluation addressed the following questions:

Relevance

1. Is there a continued need for CFS?
2. To what extent are CFS objectives aligned with the priorities of government and the strategic outcomes of ISED?
3. To what extent does CFS align with the roles and responsibilities of the federal government? Does CFS complement, duplicate or overlap other government programs or private services?

Performance

4. To what extent did the program refurbish and distribute surplus computer equipment to beneficiaries and increase accessibility to computer technology for Canadians?
5. To what extent has the program contributed to learning opportunities for youth of diverse backgrounds and genders?
6. To what extent has the program enhanced the employability of youth through ICT internships?
7. How has the development of a network of partners contributed to the achievement of the programs expected outcomes?
8. To what extent has the program reduced the environmental footprint associated with surplus electronic equipment?
9. To what extent does the program demonstrate efficiency and economy?

2.3 Data collection methods

Document Review

The document review provided an understanding of CFS, the alignment with government priorities and its achievement of expected outcomes. Key documents included program foundational documents, contribution agreements, project summary forms and recipient progress reports; and government priority-setting documents such as Budgets and Reports on Plans and Priorities.

Literature Review

The literature review addressed the core evaluation issues of continued need and federal roles and responsibilities. It examined the environmental impacts of electronic waste (e-waste) and the benefits of re-use and recycling. The literature review also looked at the continued need to provide work experience to youth of diverse backgrounds and genders.

Interviews

A total of 44 interviews were conducted to gain qualitative information related to relevance and performance. Participants included the following types of respondents^{Footnote 4}:

• ISED management and staff (4)
• Other government departments (2)
• Eligible recipients (14)
• Workshop managers (3)
• Board of Directors of recipient organizations (3)
• Voluntary sector (4)
• Network of partners (5)
• Subject matter experts on education and/or the environmental impact of e-waste (2)
• Program beneficiaries (5)
• Provincial representatives (2)

Youth Survey

A web-based survey of TWEP and other youth participants was conducted to assess the youth participants' perspectives on the continued need of the program. The survey assessed the work experience and its usefulness in terms of practical experience, technical training and skills development, as well as the employability of youth participants.

In total, 535 youth^{Footnote 5} from across Canada were invited to complete the survey A total of 197 youth completed the survey, representing a response rate of 37% An additional 23 youth participated in group interviews thus representing the views of 220 youth participants.

Site Visits

There are about 30 workshops across Canada that deliver CFS. A sample of three locations was selected in consultation with the program area and included the National Technology Centre in Gatineau, Renewed Computer Technology in Mississauga and Computers for Education Technology New Brunswick Limited in Fredericton. Site visits allowed the evaluation team to familiarize themselves with the logistics of the program through demonstrations and walkthroughs, and conduct in-person interviews with workshop managers, youth and volunteers.

Administrative Data Analysis

Operational and other program data collected by CFS was used to assess performance. Specifically, the evaluators reviewed eligible recipients' quarterly and semi-annual reports as well as program roll-ups of these reports.

Financial Data Analysis

A high-level financial analysis of CFS was conducted to address the evaluation issues of efficiency and economy. The allocation and utilization of resources were also reviewed

2.4 Limitations

The following were the data limitations:

Data quality

There were several discrepancies with the performance data reported by eligible recipients. For example, full sections of quarterly and semi-annual reports were either incomplete or duplicated from a previous reporting cycle, making it a challenge for evaluators to assess program outcomes from the data. There were also discrepancies between the number of computers distributed over the evaluation period compared to the total number of computers distributed to each of the various organizations. To address this, the evaluators met frequently with program staff to review and adjustments were made to ensure an accurate information base.

Calculating cash and in-kind contributions

The program is decentralized and operational models vary among eligible recipients depending on the resources available. As mentioned earlier, one eligible recipient may have to pay rent and adapt by utilizing more volunteers, while another may have secured donated space and be able to retain more paid staff. Eligible recipients are required to report on cash and in-kind donations^{Footnote 6} they receive on an annual basis. However, interviewees indicated that it is difficult to accurately calculate the dollar value of in-kind donations because they are numerous and varied. Some expressed that they probably underreport when calculating the in-kind amount. As a result, the evaluators were unable to accurately calculate program leveraging over the evaluation period.

Contact Information for Youth Survey

Eligible recipients are not required to maintain contact information for youth participants who have completed their work terms so evaluators contacted eligible recipients to request this information. A total of 747 youth names were provided. Of these, 502 contacts included an email address (67%). Youth participants with telephone contact information and no email address were contacted by telephone to request email addresses and to encourage them to participate in the survey. Of the 192 calls made, 56 youth participants provided their email addresses. An introductory email determined that 535 e-mail addresses were viable.

3.0 Findings

3.1 Relevance

3.1.1. Is there a continued need for CFS?

The continued need for the program was assessed by examining the objectives of the CFS, which are to support the provision of computers to beneficiaries, to help provide work experience and ICT skills development for youth and to reduce the environmental footprint associated with surplus computer equipment.

Computers to beneficiaries
A 2015 Organization for Economic Cooperation and Development (OECD)^{Footnote 7} report states that ICT has revolutionized virtually every aspect of life and work and students who are unable to navigate the complex digital landscape will no longer be able to participate fully in the economic, social and cultural life around them. The report further explains that although students need to acquire basic skills in reading and writing to navigate the digital landscape, technology is the only way to dramatically expand access to knowledge.

A 2014 report^{Footnote 8} by the International Telecommunication Union (ITU) echoes the benefits that ICTs have brought to the education sector including broadening the availability of quality educational materials and resources. In the words of one interviewee and acknowledged by others, "Technology is a necessary part of Canadians' access to services, and participation in our society and economy." Interviewees indicated that the program provides computers to those who could not otherwise afford them.

In 2015, the program's mandate was expanded to include not-for-profit organizations across Canada. To help refugees settle into life in Canada the program delivered 7,547 computers to Syrian families as part of Immigration, Refugees and Citizenship Canada's #WelcomeRefugees program.

Previously, CFS delivered computers into the institutional setting of schools and other learning organizations In that environment, Internet connectivity and skills transfer were available for students. The broadening of the program's mandate, and particularly the participation in the #WelcomeRefugees program, saw computers going to families, potentially without Internet connectivity or the skill set to access the technology or use it to its potential. A recent ISED report^{Footnote 9} asserted that Internet connectivity is both increasingly necessary to effectively use a computer and that Internet access is a significant issue for many low income households. With an expanded mandate, newly identified needs are emerging for those using CFS computers.

Work experience for youth
A 2011 report^{Footnote 10} prepared by the Université de Montréal found that workplace training associated with the reuse sector generates benefits for youth struggling with exclusion or academic underachievement. These youth can benefit from professional training and an initial paid work experience. Such skills enable youth to integrate or reintegrate into the labour market or the school system and, for some, to free themselves from economic dependence. Further, the 2015 summative evaluation of the Horizontal Youth Employment Strategy found that there is a demonstrable need to assist Canadian youth in finding and maintaining employment.

The youth survey conducted for this evaluation demonstrates that 88% of the surveyed youth strongly agree or agree (65% and 23%) that there is a need to continue providing this kind of experience for youth. In addition, interviewees explained that the CFS program provides youth with knowledge and experience, both valuable tools when entering the workforce.

Reduced environmental footprint
Since 2004, electronics recycling programs across Canada have diverted over 500,000 tonnes^{Footnote 11} of end-of-life electronics from landfill^{Footnote 12}. Electronics Product Stewardship Canada (EPSC) reports that Canada is a world leader in electronics stewardship. At over five kilograms, Canada's per capita volume of comparable recycled electronics surpasses that of the United States, as well as many European countries.^{Footnote 13}

A review of academic literature pertaining to the environmental benefits of refurbishing computers explains that diverting electronics goods from landfills and incinerators prevents the release to the environment of toxic materials that are present in these goods.^{Footnote 14Footnote 15}. E-waste contains precious and special metals, including gold, silver, palladium and platinum, as well as potentially toxic substances such as lead, mercury, cadmium and beryllium. Therefore, responsible end-of-life management of e-waste allows the recovery of valuable components and the effective e-management of hazardous and toxic components.^{Footnote 16}

The materiel footprint of the electronics industry is shrinking rapidly with lighter weight products.^{Footnote 17} Nonetheless, refurbishing computer equipment that still carries value leads to an extended product's life, resulting in reduced material and energy consumption for the production of new computers.^{Footnote 18} The majority, estimated to be 80%, of the life cycle energy for computers is used in the manufacturing phase ^{Footnote 19} Research conducted at the University of Montreal concludes that the increase in impacts during the use phase of refurbished equipment is negligible compared to the avoided impacts associated with the production of new equipment.^{Footnote 20} Results from this research demonstrate greater social, economic and environmental benefits for reuse of computers.^{Footnote 21} A 2010 report^{Footnote 22} states that reusing working computers is up to 20 times more energy efficient than recycling them.

Further, studies^{Footnote 23} have shown that refurbishing computers has a positive environmental impact, such as energy conservation, greenhouse gas reduction, solid and hazardous waste reduction, reduced air and water emissions as well as other environmental benefits. When refurbishing is not feasible, recycled computers reduce use of landfill space, create less toxic chemical emissions and can be used as raw material for other products. Both refurbishing and recycling computers ultimately reduces their environmental footprint.

Interviewees explained that CFS gives governments and businesses the opportunity to have their surplus computers reused, thus protecting the environment by giving computers a second life.

3.1.2. To what extent are CFS objectives linked to the priorities of government and the strategic outcomes of ISED?

Through Canada's Inclusive Innovation Agenda, the Government plans to harness the digital economy across sectors to encourage digital adoption and strengthen competitiveness. Competing in a digital world, putting Canada at the forefront of economy-wide digital development and adoption, is an area for action in the agenda As such, the Government is looking for innovative ways to develop stronger digital skills among Canadians, making access to computers necessary.

The Government is also committed to helping young Canadians gain the skills, abilities and work experience they need to find and maintain good employment The Youth Employment Strategy (YES), coordinated by ESDC, is the Government's key labour market program to help young people Budget 2016 provided an additional investment to YES of $165.4 million for 2016-17 and 2017-18, in part to fund Green Jobs for Youth.^{Footnote 24} Youth hired by CFS, through TWEP funding, gain hands-on training in computer refurbishment as well as develop soft skills Due to the nature of the CFS program, all TWEP internship positions are considered "Green Jobs."

The Federal Sustainable Development Strategy 2016-2019 aims to protect Canadians and build healthy communities To achieve this priority, the Government will "participate in joint initiatives to manage risks posed by harmful substances to nature and water, and work with domestic and international partners through programs like Computers for Schools" CFS provides an environmentally responsible solution for disposing of surplus government computers.

The CFS program contributes to the strategic outcome: Canadian Businesses and Communities are Competitive The program is designed to support this outcome by increasing the accessibility of digital tools needed for the skills of tomorrow and by providing internship opportunities for young Canadians to gain valuable experience in the field of information and communications technologies.^{Footnote 25}

3.1.3 To what extent does the CFS align with the roles and responsibilities of the federal government? Does CFS complement, duplicate or overlap other government programs or private services?

The CFS program helps meet federal roles and responsibilities with respect to the disposal of crown assets and sustainable development The 2011 Government of Canada Guide to Management of Materiel (Section 3.4.2) states that Departments must offer right of first refusal of all surplus personal computers, laptops, servers, monitors, keyboards, mice, printers, modems, hubs, network cards, hard drives, etc., to ISED's Computers for Schools Program.

According to the Federal Sustainability Act of 2008, each Department is responsible for preparing a sustainable development strategy that contributes to the Federal Sustainable Development Strategy. ISED's 2016-19 Sustainable Development Strategy indicates that the department will continue to co-operate with partners across Canada to implement the CFS program to divert equipment from landfills thus protecting nature, preventing water pollution (through seepage of toxins from discarded equipment) and providing economic and social benefits to Canadians CFS plays a unique role in supporting these requirements by providing a government-wide approach to ensure that significant quantities of surplus computer equipment are collected, refurbished and distributed to schools and other beneficiaries on a national level.

Additionally, the objectives of CFS align with the roles and responsibilities of the federal government under the Department of Industry Act of 1995 According to this legislation, the powers, duties and functions of the Minister of ISED extend to matters relating to "industry and technology in Canada," and the Minister may exercise those powers to "strengthen the national economy and promote sustainable development" and "foster and promote science and technology in Canada."^{Footnote 26}

The CFS program also plays a role in ensuring the protection of personal and other information on government computers With regards to purging information from the hard-drive and data storage devices, Section 3.4.1 of the 2011 Government of Canada Guide to Management of Materiel states that, "Departments are responsible for ensuring that security concerns are addressed." Although it is the responsibility of federal departments and agencies to cleanse all donated computers of classified and protected information prior to the release to CFS, the program provides additional assurance by wiping hard drives a second time All eligible recipients confirmed that a hard drive wiping procedure is conducted in accordance with standards set by the Royal Canadian Mounted Police, Communications Security Establishment Canada, and/or the Electronic Products Recycling Association.

Evidence collected for this assessment also suggests that the role of the federal government is still key to the overall success of the program A KPMG Operational Review (2013) found that eligible recipients believe the partnership with the federal government brings significant credibility to both program and recipient organizations, particularly when seeking to establish partnerships with other organizations As such, the presence of the CFS Program Office with dedicated resources to manage the national program is perceived as essential for continued strength of the brand, consistency in the delivery of the program's mandate, and establishment of national policies and procedures.

Interviewees echoed this, underscoring the importance of the federal government's role in the program It allows for federal leadership on environmental issues and raises awareness regarding the benefits of reuse and recycling It also provides access to technology, skills development and employment for youth.

The CFS program is unique and welcomes donations from sources other than the federal government, including provinces, territories and for-profit corporations The literature review and interviewees confirmed that there is no known comparable program within Canada Interviewees indicated that there may be some organizations that refurbish computers but none operate on a national scale, nor do they have the same objectives as the CFS program. Generally, the organizations mentioned were local private enterprises that sell refurbished computers, but concerns were raised from interviewees with regard to quality control and the disposal of environmental waste of some of these companies.

3.2 Performance

3.2.1 To what extent did the program refurbish and distribute surplus computer equipment to beneficiaries and increase accessibility to computer technology for Canadians?

During the evaluation period, there were 369,573 computers refurbished and distributed to beneficiaries Figure 2 below shows the distribution of CFS computers by type of organization during this same timeframe Similar to the last evaluation of this program, the majority of computers (78%) were delivered to schools.

Figure 2: Distribution of Computers by Type of Organization

ISED's 2015–2016 RPP sets a target of 70,000 for the number of refurbished computer units delivered annually. Over the assessment period, the program achieved this target with an annual average of 73,915 refurbished computer units delivered, a 13.5% increase from the previous evaluation period.

Interviewees indicated that the program gives computer access to organizations, schools and individuals who would not otherwise be able to afford the computers, increasing accessibility to computers for Canadians They further suggested that the expanded CFS mandate (Budget 2015) will help the program reach even more vulnerable groups.

3.2.2 To what extent has the program contributed to learning opportunities for youth of diverse backgrounds and genders?

During the evaluation period there were 1,495 TWEP interns (approximately 300/year) employed in CFS workshops across Canada. This exceeds the target of 250 youth interns gaining work experience in the information and communication technologies field, as shown in ISED's 2015–2016 RPP The assessment found evidence that other young people were also profiting from the program, including those participating through co-op programs and other student internships.^{Footnote 27}

The youth survey found that 82.2% of respondent's self-identified as male, 15.5% as female and 2.3% as other This was corroborated by all eligible recipient interviewees who indicated that most of the TWEP participants are male.

A 2015 report^{Footnote 28}, initiated by the program to help understand the barriers to female participation in TWEP, found that "while women are under-represented in the IT field in Canada among both employees and students, they are even more under-represented among CFS TWEPs."^{Footnote 29} The report highlights that opportunities exist to increase the number of women taking part in the TWEP program and provides many recommendations that include targeted outreach, allowing for flexible work arrangements, as well as the removal of "heavy lifting" as a requirement in job postings Program staff and eligible recipient interviewees mentioned that the removal of the heavy lifting requirement from recruitment posters has resulted in more young women applying to the program.

CFS would profit from further studying their intern and co-op student population to ensure there are no barriers for other groups to participating in the program.

3.2.3 To what extent has the program enhanced the employability of youth through ICT internships?

The majority of respondents to the youth survey felt that their time at CFS helped them gain the skills they need for their career prospects (35% fully agreed, 49% partially agreed) and that the program helped inform decision-making about their future careers (33% fully agreed, 48% partially agreed) Of the survey respondents who completed an internship, 42% went to work in an IT-related area, 30% went back to school, 16% went to work in a non-IT-related area and 12% did other activities In the words of one survey respondent, and echoed by others, "I think it was a great kick start in my computer/IT career." Many survey respondents expressed the importance of relevant work experience in the IT field and indicated that the CFS program is often their first opportunity to attain such training.

Interviewees also indicated that the program contributes to skills development and work experience for participants There was widespread agreement amongst interviewees that the TWEP program is often the first hands-on job experience for many youth participants and that it provides an opportunity to teach and equip youth for their careers. Interviewees explained that TWEP interns gain valuable technical experience and skills, as well as soft skills (e.g time management and good work habits) that contribute to their overall learning experience.

It is of note that eligible recipients are not required to maintain contact with youth participants who have completed their work terms and therefore the evaluators had limited data to measure the employability of youth participants after completion of their internship with CFS Maintaining contact information post-employment and sending a follow-up questionnaire a year following the conclusion of an internship may yield more robust information.

3.2.4 How has the development of a network of partners contributed to the achievement of the programs expected outcomes?

Over the years CFS has built a wide range of partnerships that provide donations in-kind contributions and other supports to the program These partnerships are vital to the overall success of the program Using the figures reported annually by eligible recipients, Table 1 below shows that over the evaluation period, partner contributions represent approximately 34% of eligible recipients' funding^{Footnote 30}.

The program has existing partnerships at the national level that provide transportation and software/hardware For example, the Canadian National Railway Company has been a long-time partner of the program and has transported CFS computers across Canada for over 20 years for free. Also of note, Microsoft Corporation has donated computer licences over the years and most recently donated 7,700 licences to the #WelcomeRefugees project.

There are also local partnerships between eligible recipients and other private and/or public entities Eligible recipients have their own network of partners that facilitate the delivery of the program in their area. Support varies from location to location and includes, for example, cash, donated workshop space, distribution networks, transportation of equipment and volunteer workers.^{Footnote 31} For instance, one eligible recipient works with a local organization supporting new immigrants The organization sets up a training room with computers, the partner organization brings people in and offers them three days of computer training, after which the participants take the computer home for their use.

In addition, eligible recipients receive computer donations from a number of sources Over the course of the evaluation period, the federal government provided 44% of the donated computers to the program and eligible recipients secured 56% of their computer equipment donations from their network of partners.

Interviewees (program staff and eligible recipients) indicated that partners make a significant contribution to the program In the words of one interviewee, and acknowledged by others, "without partners, CFS would not be the program that it is today, they are a big help."

At both the national and local levels, the network of partners contributes to the achievement of the programs' expected outcomes. Donations of computer equipment, from all sources, increases the number of computers that can be distributed to beneficiaries, reduces the environmental footprint by giving surplus electronic equipment a second life, and provides youth with more opportunities to gain work experience Interviewees noted that other forms of in-kind contributions from partners, including cash, free transportation, licences, rent and volunteer work help make the program sustainable and improve program impacts.

3.2.5 To what extent has the program reduced the environmental footprint associated with surplus electronic equipment?

Interviewees explained that CFS is important as it gives governments and businesses the opportunity to have their surplus computer equipment reused or recycled, thus reducing the environmental footprint. Once donated, equipment is either refurbished or recycled, depending on quality and demand of donated units Program data demonstrates that over 1.4 million computers, monitors and printers were donated to the CFS program over the evaluation period.

A 2010 report^{Footnote 32} prepared by Computer Aid International concludes that the presence of hazardous materials such as lead, nickel and mercury pose risks to human health or the environment if improperly handled so keeping them out of the landfill makes sense As well, high levels of product replacement and the concentration of energy intensity in the ICT production rather than use phase (80 and 20 percent, respectively) means that any activity that extends the life of computers, such as reuse, should be prioritized This same report states that reusing working computers is up to 20 times more energy efficient than recycling them.

Although no interviewees could confirm, most said refurbished computers have a second life span of three to five years Eligible recipients explained that equipment that is not reused is picked up by certified e-waste recyclers, thus avoiding landfill Table 2 below shows that the percentage of units sent to recycling between the current evaluation and the previous evaluation has remained fairly constant, with the exception of printers.

The program suggests the number of printers being sent to recycling over the evaluation period may be the result of the federal government's printer optimization work, which has led to additional printers being surplused Interviewees indicated that in some cases the demand for printers is lower because they are leased by schools rather than purchased.^{Footnote 33}

3.2.6 To what extent does the program demonstrate efficiency and economy?

The efficiency and economy of CFS was assessed by analysing program expenditures over the evaluation period, changes in cost per refurbished computer, leveraging, and efficiencies gained since the previous evaluation A review of actions taken toward addressing the recommendations from the 2012 evaluation was also undertaken.

Expenditures
CFS expenditures totaled $42.1 million over five years (2011–12 to 2015–16), of which $3.3 million was for operating expenditures and $38.8 million for Grants and Contributions Figure 3 below shows that Salary and Operations and Maintenance (O&M) expenditures have decreased 29% over the evaluation period ^{Footnote 34} while the CFS program continues to meet its target of 70,000^{Footnote 35} refurbished computers delivered annually to beneficiaries, suggesting program efficiencies.

Figure 3: Total salary and O&M expenditures by year, CFS and TWEP combined.

Cost per refurbished computer
Over the period of the evaluation, 369,573 computers were refurbished for a total cost of $38,808,207^{Footnote 36} This represents a cost of $105 per refurbished computer^{Footnote 37}, well below that of a new unit at market value Findings from the previous evaluation show a cost of $130.89 per refurbished computer When compared to the current program cost of $105, program efficiency has improved since the previous evaluation.

Leveraging
One measure of a program's efficiency is its ability to leverage funding from additional sources Leveraged funds, including both cash and in-kind donations, such as software, transportation and rent, are gained by the CFS program through national and local partnerships While the program does not set specific targets for leveraged funds, eligible recipients are asked to estimate in-kind and cash donations in their CAs with ISED, as well as to report on all donations in their final semi-annual report to ISED. For every dollar invested in the program over the course of the evaluation period, about 52 cents was leveraged from other sources, without counting the value of the donated computer equipment.

Other evidence of efficiencies since the previous evaluation
The evaluators found additional evidence that the program has worked to realize efficiencies since the previous evaluation.

In 2015 the CFS program introduced the 2015-2018 multi-year CAs. Previously, the program renewed CAs on an annual basis Eligible recipient interviewees remarked that the introduction of multi-year CAs has saved them time by providing greater stability to the program, allowing them more time on program activities.

Additionally, the program altered reporting in 2014-15 Though quarterly reports are required under the Terms and Conditions, the program office reduced the requirements of the first quarter and third quarter reports to two numbers: TWEPs hired and computers distributed, thus reducing the reporting burden for eligible recipients With this reduction, eligible recipients spend less time providing information, which gives them more time to concentrate on the day-to-day management of their refurbishing workshops.^{Footnote 38}

Finally, the 2012 evaluation recommended updating the program's performance measurement system, as well as developing a common database The logic model and performance indicators were revised in conjunction with updating the program's Performance Measurement Strategy in 2012 and again in 2016 in preparation for this assessment. However, the program continues to mine data manually, resulting in data quality issues and some challenges with program reporting.

4.0 Conclusions and recommendation

4.1 Relevance

• There is a continued need to provide refurbished computers to students and other Canadians. CFS provides an environmentally responsible means for governments and businesses to dispose of surplus computers. Additionally, providing youth with internships gives them opportunities to develop the skills necessary to enter the workforce.
• CFS aligns with federal responsibilities to foster access to technology, maximize the use of crown assets and contribute to sustainable development through the appropriate disposal of IT equipment The CFS program does not duplicate or overlap any other government program.
• The objectives of the program are consistent with federal government priorities related to developing stronger digital skills among Canadians, providing work experience to youth and reducing the Government's environmental footprint.

4.2 Performance

• CFS has distributed nearly 370,000 refurbished computers to beneficiaries over the past five years, with the majority delivered to schools In addition to providing technology to schools and assisting not-for-profit organizations, the recent program expansion gives access to computer technology to Canadians who could not otherwise afford it.
• With respect to work experience, CFS enhances the employability of youth by providing hands-on experience in the IT field and assists in the development of both ICT and soft-skills.
• About 300 interns were employed annually in CFS workshops across Canada, exceeding annual published targets The program has contributed to learning opportunities for youth There is an opportunity for the program to recruit additional female participants.
• The network of partners surrounding CFS brings with it substantial in-kind and cash donations These donations enable the program to operate and succeed.
• The program reduces the environmental footprint of government and businesses through reuse and recycling of their computer equipment.
• The program demonstrates economy and efficiency and continues to meet delivery targets despite a reduction in resources over the assessment period However, the program continues to mine data manually, resulting in data quality issues and some challenges with program reporting.

4.3 Recommendation

The findings of the evaluation led to the following recommendation:

1. The CFS program should consider modernizing its data collection, capture and storage with a view to ensuring adequate performance information is available.
2. The CFS program should continue to explore the diversity of its interns and consider what more could be done to attract female candidates.

Management Response and Action Plan

A - For inclusion in the report

The findings and recommendations of the Computers for Schools Evaluation Report were presented to program management. Management has agreed with the findings included in this report and will take action to address all applicable recommendations by March 2018.

B - For follow-up purposes - Detailed actions to address the recommendations in the report