Jump to ContentJump to Main Navigation
Bulk CollectionSystematic Government Access to Private-Sector Data$

Fred H. Cate and James X. Dempsey

Print publication date: 2017

Print ISBN-13: 9780190685515

Published to Oxford Scholarship Online: October 2017

DOI: 10.1093/oso/9780190685515.001.0001

Show Summary Details
Page of

PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. All Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. date: 23 January 2021

(p.xxv) Introduction and Background

(p.xxv) Introduction and Background

Bulk Collection

Fred H. Cate

James X. Dempsey

Oxford University Press

The tensions between privacy and security seem sharper than ever. Concerns about terrorism are driving many governments to adopt more expansive surveillance powers, while human rights courts, at least in Europe, continue to cite privacy rights to strike down overbroad measures. The digital services woven into our personal and professional lives generate more and more information revealing our movements, actions, and intentions, while encryption that shields communications from interception and blocks access to data stored on mobile devices is becoming widespread. Big data techniques make it easier for governments to ingest large amounts of data and mine it to discern patterns and make decisions, but governments simultaneously complain they are “going dark” in the face of technological change, unable to obtain evidence crucial to criminal and national security investigations. Regulators seek to promote enhanced cybersecurity, yet fairly simple phishing techniques expose huge volumes of email and documents to hackers, undermining not only privacy but the democratic process.

This volume represents the culmination of a nearly six-year project examining this tension. It began as an effort to obtain a snapshot of what seemed to be growing government demands for bulk access to data held by the private sector. After leaks and authorized disclosures lifted the shroud of secrecy around the bulk collection activities of some governments, it turned into something much more ambitious: an effort to explore what should be the rules for government access to data and what should be the responses of private sector companies to those demands.

Throughout, the project unfolded in the context of the vast changes wrought by the ongoing revolution in information and communications technology. As a part of daily life, individuals around the world use services that collect and store data in digital form. The expansive aggregation of personal data in the hands of private-sector companies is true equally of businesses firmly rooted in the physical world—retailers, health care providers, financial institutions, utilities, airlines, hotels—and of those based online. The emergence of the Internet of Things—always on, always collecting—is further amplifying this trend.

(p.xxvi) Within this ocean of data is information of value to governments pursuing legitimate interests and, of course, to those seeking to suppress and control. Governments understandably want access to this data. At the top of their list is communications data—the content of communications and also records of who is calling whom, mobile phone location data, and Internet connection records. Also of interest are bank records, travel records, and potentially any kind of data that could reveal a person’s activities. Essentially every government in the world claims the power to compel disclosure of this data by the companies that hold it. The rules surrounding such disclosures—how much can be obtained, under what standard, and upon the approval of what authority—remain an urgent concern of both citizens and the companies holding their data.

Our project was premised on the view that there is a fundamental distinction between situations where government agents demand from third parties data regarding a particular target and, on the other hand, situations where the government is collecting large quantities of data without discrimination. For the former, which traditionally characterized law enforcement investigations, practices and rules have for some time been relatively clear (even as the variety of information available has expanded): when seeking data about an individual in a criminal investigation, government agents must have some threshold of particularized suspicion linking that person to a specific crime, they must obtain independent authorization for the surveillance or data acquisition, and the intrusion on privacy must be limited in time and scope to the acquisition of evidence relevant to the crime being investigated.

However, it is now clear, governments have also been collecting information without particularized suspicion, often for intelligence or national security purposes but also, almost unnoticed, for regulatory purposes. These non-particularized, bulk demands pose unique questions that our project explored. Four issues in particular are salient. The first concerns transparency: What powers are governments exercising? When we began this work, bulk collection programs conducted in the name of national security had not been publicly avowed. The second question is about legality: Does a publicly-available statute authorize and define the government’s power in clear terms? The third issue is normative: What standards should limit government access, and what structure of control and oversight can assure against abuse? Finally, even if publicly avowed and even if statutorily authorized, can a system of safeguards and oversight ever be robust enough to legitimize mass surveillance, or are bulk programs incompatible with human rights principles of necessity and proportionality?

Our Process

In 2011, under the auspices of The Privacy Projects, we began exploring what we called at the time “systematic government access to data held by the private sector.” By “systematic access,” we meant both direct access by the government to private-sector databases, without the mediation or interaction of an employee or agent of the entity holding the data; and government access, whether or not (p.xxvii) mediated by a company, to large volumes of private-sector data. It seemed to us at the time that there had been an increase worldwide in government demands for data held by the private sector, driven by a variety of factors, and that this had included an expansion in government requests for direct access or bulk disclosures.

Two years before the Snowden leaks, we commissioned papers from leading experts in nine countries (Australia, Canada, China, Germany, India, Israel, Japan, the UK, and the United States), asking them to explore what, if anything, was publicly known about bulk collection in their countries and to describe the laws regarding broad government access to private-sector data. In April 2012, we convened a meeting in Washington of academics, privacy advocates, and private-sector leaders to review those papers and chart a course for further research.2 Among other things, we decided to expand the geographic scope of the study and commissioned four additional papers (covering Brazil, France, Italy, and the Republic of Korea),3 which were the subject of another multi-stakeholder roundtable, held in London in May 2013.

These initial papers confirmed our thesis, identifying various examples of “systematic access” in a wide range of countries. The research also found a general lack of transparency about the nature and scope of data collection practices carried out in the name of national security or foreign intelligence. Many were not publicly acknowledged by the governments, and the companies subject to the demands were prohibited from disclosing them. Moreover, laws on the books did not expressly authorize bulk collection. Even the experts we enlisted admitted that they were uncertain of what the law permitted or how it was being interpreted. Oversight mechanisms, our authors found, were limited and, if they existed, were themselves often shrouded in secrecy.

In June 2013, weeks after our London roundtable, the Snowden leaks began. Unauthorized and authorized disclosures of intelligence programs in the United States, the UK, and some other European countries partly lifted the shroud of secrecy, at least with respect to some countries. The disclosures gave detailed substance to our core concerns about expansive and lightly regulated government demands for access to data held (or transmitted) by the private sector. “Bulk surveillance” came to be featured prominently in national and international debates over governmental power, corporate responsibility, and individual privacy. Policymakers around the world professed shock and concern about the intrusiveness of government (usually other governments’) programs of bulk collection.

In the immediate wake of the Snowden leaks, however, much of the commentary was misleading, especially in suggesting that bulk collection was (p.xxviii) predominantly a US and UK practice. Our earlier research had shown that the practice was much more widespread. To highlight our findings and to seek to drive a more accurate discussion of the legal and policy issues, The Privacy Projects organized a public workshop in Brussels in November 2013 for private-sector and civil society representatives to meet with data protection authorities and other government officials. The Privacy Projects also commissioned a major article summarizing the project’s findings to date.4 We also turned our attention to the questions of oversight and accountability, hosting additional workshops in 2014 in Montreal and London focused on means of achieving accountability when the government accesses private-sector records.

Finally, in an effort to pull together these various threads, we commissioned a series of essays from prominent industry leaders, activists, and academics from around the world. These papers addressed in practical terms the elements of oversight that should be applied to any government program seeking broad access to personal data held by the private sector. Other papers address the question of how industry should respond to such requests or demands and how the divergent interests of government, companies, and individuals can be understood. Last, we commissioned papers that assessed bulk or indiscriminate collection against the evolving framework of international law and human rights law.

Our Findings

This volume contains the fruits of our project. Twelve country reports have been complied here. Most of them have been updated to account for new revelations, laws, and court decisions. They are accompanied by the comparative analysis of Ira Rubinstein, Greg Nojeim, and Ron Lee, also updated. They provide extensive evidence that governments around the world have been collecting data on a very large scale. These collection programs are often conducted in the name of national security, but some are also available for ordinary law enforcement, and there are many broad collection programs conducted for regulatory purposes, such as tax compliance.

The country reports show that, despite some reforms, the worldwide trend continues in the direction of ever larger collections. Indeed, the only country that has conclusively terminated a bulk collection program in recent years is the United States. Counter to its Snowden-induced reputation as a voracious collector of data, in 2015, the United States ended the bulk collection of metadata on domestic and international calls. Congress enacted the USA FREEDOM Act, which amended all potentially applicable statutes to make it clear that they could not be used as the basis for bulk domestic collection in national security matters. Meanwhile, the UK, France, Germany and other countries have ratified or expanded collection programs.

(p.xxix) Many of the country reports discuss not only programs of bulk or mass surveillance—surveillance that involves, for example, all telephone calls or all Internet service—but also programs that are targeted (focused on specific individuals or accounts) but that nevertheless collect very large amounts of data on large numbers of individuals. Given modern technology, even targeted collection programs can be very broad. The intake of such programs, if stored for extended periods of time, can constitute quite a comprehensive database on quite a large swath of the public. How such data is searched, for example, may be as important as the rules for how it was collected in the first place. Even though our baseline distinction between targeted and indiscriminate collection remains valid, the country reports remind us that it is probably best to view government collection activities as arrayed across a spectrum from the tightly targeted and rarely applied to the targeted but broadly applied to the comprehensive. Especially where companies are required to maintain databases of records (data retention mandates) and to install filtering or retrieval capabilities on their networks for use by the government at will (as France and the UK now seem to require), the distinction between targeted and bulk collection may disappear. Systematic access (our initial focus) may no longer require bulk collection.

The country reports and the papers in the second half of the volume also reveal that there have been some positive developments since we began this project. Although powers of bulk surveillance had, until recently in all the countries surveyed, been exercised in the dark, lately there has been a move toward greater transparency. In response to the Snowden leaks, the United States and the UK officially acknowledged a number of practices. In other countries, bulk collection programs continue to be shrouded in secrecy, but there has been “progress” in the sense that a number of countries have amended their laws to more explicitly describe the powers exercised by their governments. This at least theoretically subjects the programs to the democratic process.

Another positive development is that these new laws, while generally ratifying or even extending bulk collection powers, have included new oversight or accountability measures. The UK’s new Investigatory Powers Act includes a “double-lock” for the most intrusive powers, so that warrants issued by a Secretary of State will also require the approval of a senior judge. The Act creates a new Investigatory Powers Commissioner to oversee how the new powers are used, establishes limits on government access to journalistic and legally privileged material, and creates new criminal offenses for misusing the powers. France, in its 2015 law, created a new, independent Commission for Oversight of Intelligence Gathering Techniques. Under the law, intelligence gathering measures can be implemented only when a specific authorization is given by the prime minister or his or her designee, and the prime minister’s authorization can be granted only after the Commission has rendered an opinion, albeit one that is not binding, on the compatibility of the measure with the principles set forth in the law.5

(p.xxx) Several chapters in this volume explore the development of oversight mechanisms. With Marty Abrams, we have a chapter showing how the principle of accountability, now woven into data protection law in the commercial context, has direct application to government surveillance. Eduardo Bertoni and Collin Kurre describe still-evolving oversight mechanisms in Latin America. Nico van Eijk, drawing on the jurisprudence of the European Court of Human Rights, fleshes out the multiple elements needed for a truly effective oversight program.

As van Eijk explains, effective oversight must encompass prior authorization, after-the- fact review, and redress of complaints. No one body or structure can be relied on to provide adequate control of government surveillance. Courts, no matter how independent, can secretly approve programs that seem unreasonable in the light of day. Parliamentary bodies may grant broad powers. Effective oversight can be achieved only with a web of checks and balances, implemented by multiple bodies of varying competencies, reinforcing each other. Overall, the principles of oversight and accountability seem to be gaining wide credence in democratic countries, if only because governments recognize that they must maintain some level of trust if they are to retain their expansive powers.

But the most remarkable development of the past six years, second only to the startling revelations of bulk collection, has been the insistence of human rights courts and other institutions on the principles of privacy and the willingness of those bodies to strike down or criticize surveillance measures even when justified in the name of fighting terrorism. Especially assertive have been the two human rights courts in Europe: the European Court of Human Rights (ECtHR) and the Court of Justice of the European Union (CJEU). In the Schrems case, the CJEU invalidated the EU-US Safe Harbor for failing to address standards for US government access to data that global companies transfer from Europe to the United States for storage and other processing. In Digital Rights Ireland, it overturned the EU directive that had required service providers to retain metadata on customer communications. The ECtHR invalidated surveillance laws in Russia (the Zakharov case) and Hungary (the Szabó and Vissy case) on the ground that the laws were insufficiently discriminate in their targeting standards. At the national level, the French Constitutional Council in October 2016 declared a provision of the 2015 French law unconstitutional. Also in October 2016, the UK’s investigatory powers tribunal ruled that British intelligence agencies had been unlawfully collecting massive volumes of confidential personal data without proper oversight for 17 years.6 Nonjudicial independent oversight bodies also proved their value. In the (p.xxxi) United States, the Privacy and Civil Liberties Oversight Board played an important role in ending the program that collected telephone calling records in bulk.7

As the chapters by Ashley Deeks and Sarah St.Vincent, as well as the comparative analysis of Rubinstein, Nojeim, and Lee, show, there is remarkable consistency in defining the components of an effective system of checks and balances. The elements of the framework of oversight and accountability are drawn from long-accepted principles of the rule of law, human rights, and democratic governance. Most important for our project, the conclusion that bulk or indiscriminate collection is fundamentally incompatible with human rights principles may be gaining hold.

Two actions taken after most of the chapters in this book were written—the UK’s November 2016 adoption of a new investigatory powers act and the December 2016 decision of the CJEU striking down national data retention laws of Sweden and the UK—illustrate both the assertion of bulk powers by governments and the application of human rights principles to reject those claims.

The UK’s Investigatory Powers Act lays out a breathtaking array of surveillance powers. It authorizes the issuance of notices to communications service providers requiring them to retain data on the activities of all users. Government authorities will be able to access this data using a process called the “request filter.” Described by the Act’s proponents as a safeguard intended to ensure that the government obtains only relevant data, the request filter also serves as a federated search engine, allowing searches across multiple corporate databases without the need to ingest them into government coffers. On top of that, the Act unabashedly embraces the concept of bulk collection, explicitly authorizing the issuance of “bulk interception warrants” for the interception of communications between persons in the UK and persons overseas; “bulk acquisition warrants,” which require telecommunications operators to disclose communications data (metadata); “bulk equipment interference warrants,” which allow hacking to obtain “overseas-related” communications or information; and “bulk personal dataset warrants,” authorizing intelligence services to retain and examine datasets where most of the information pertains to persons not, and who are unlikely to become, of interest to the intelligence service in the exercise of its functions.

Five weeks after the UK adopted its Investigatory Powers Act, the CJEU handed down its decision in the Tele2 and Watson cases, ruling invalid under EU law the Swedish data retention mandate and a similar mandate under the UK law that had preceded the Investigatory Powers Act. The Court found that even the objective of fighting serious crime cannot in itself justify national legislation providing for the general and indiscriminate retention of all traffic and location data. National legislation that covers, in a generalized manner, all subscribers and all means of electronic communication as well as all traffic data “exceeds the limits (p.xxxii) of what is strictly necessary and cannot be considered to be justified, within a democratic society.”8 The Court held that the EU directive on communications data and the Charter of Fundamental Rights of the EU “must be interpreted as precluding national legislation which, for the purpose of fighting crime, provides for the general and indiscriminate retention of all traffic and location data of all subscribers and registered users relating to all means of electronic communication.”9 The Directive and the Charter, the Court stated, “do not prevent a Member State from adopting legislation permitting, as a preventive measure, the targeted retention of traffic and location data, for the purpose of fighting serious crime, provided that the retention of data is limited, with respect to the categories of data to be retained, the means of communication affected, the persons concerned and the retention period adopted, to what is strictly necessary.”10 The Court seemed to be saying, in essence, that generalized retention (and it would seem even more so, the generalized collection) of traffic data is never permitted, since by definition it is not limited as to “the persons concerned.”

Separately, the CJEU considered the question of access to the retained data. General access to retained data cannot be regarded as limited to what is strictly necessary, it said. Instead, the national legislation concerned must be based on objective criteria in order to define the circumstances and conditions under which the national authorities are to be granted access to the data. In that regard, the Court said, “access can, as a general rule, be granted, in relation to the objective of fighting crime, only to the data of individuals suspected of planning, committing or having committed a serious crime or of being implicated in one way or another in such a crime.”11 Moreover, the Court ruled, “in order to ensure, in practice, that those conditions are fully respected, it is essential that access of the competent national authorities to retained data should, as a general rule, except in cases of validly established urgency, be subject to a prior review carried out either by a court or by an independent administrative body.”12

So, at the end of nearly six years, we are left with movement simultaneously in the direction of both more government powers and an expanded assertion of human rights principles to curtail government powers. In the digital age it is increasingly clear that governments have legitimate reasons to collect data from the private-sector entities that provide communications and other services. At the same time, the power to compel disclosure must be subject to robust checks and balances, defined by a growing international consensus around the (p.xxxiii) principles of legality, proportionality, and accountability. Even when those critical protections are present, however, it is an increasingly important and difficult question whether bulk or indiscriminate collection by the government of personal data from the private sector can ever be compatible with those principles. And it is that critical question that this volume is designed to help the reader explore. (p.xxxiv)


(2.) The first nine country reports were published in November 2012 in Volume 2, Issue No. 4 of International Data Privacy Law, https://academic.oup.com/idpl/issue/2/4.

(3.) These papers were published in February 2014 in Volume 4, Issue No. 1 of International Data Privacy Law, https://academic.oup.com/idpl/issue/4/1.

(4.) Ira Rubinstein, Greg Nojeim, and Ronald Lee, “Systematic Government Access to Personal Data: A Comparative Analysis,” 4 International Data Privacy Law 96 (2014), http://idpl.oxfordjournals.org/content/4/2.toc.

(5.) Winston Maxwell, “French Surveillance Law Permits Data Mining, Drawing Criticism from Privacy Advocates” (August 6, 2015), http://www.hldataprotection.com/2015/08/articles/international-eu-privacy/french-surveillance-law-permits-data-mining-drawing-criticism-from-privacy-advocates/.

(6.) “UK Security Agencies Unlawfully Collected Data for 17 Years, Court Rules,” The Guardian (October 17, 2016), https://www.theguardian.com/world/2016/oct/17/uk-security-agencies-unlawfully-collected-data-for-decade.

(7.) One of the authors of this volume, James X. Dempsey, served as a member of the Privacy and Civil Liberties Oversight Board. The views in this chapter and other chapters he coauthored in this volume are his own and do not represent the US government, the Board, or any Board Member.

(9.) Ibid para. 112.

(10.) Ibid. para. 108 (emphasis added).

(11.) Ibid. para. 119.

(12.) Ibid. para. 120.