The Cameras in Your Car May Be Harvesting Data as You Drive
Media Contact
Danielle Bell
SENIOR PROGRAM MANAGER FOR MEDIA RELATIONS
media@greenlining.org danielle.bell@greenlining.orgBy Keith Barry
Consumer Reports
There’s a problem with a bike lane in Brooklyn, N.Y. City officials painted it along part of Manhattan Avenue, a major thoroughfare between the Greenpoint and Williamsburg neighborhoods. But the lane suddenly ends at a busy intersection. It picks up one block east, but that isn’t obvious to cyclists. Most of them continue down Manhattan Avenue, where their path is often blocked by traffic and double-parked cars.
A few miles away on the New Jersey Turnpike in East Rutherford, N.J., there’s another problem: Drivers heading to the Meadowlands are taking a sharply curved off-ramp so fast that they have to slam on the brakes.
These and plenty of other problems around the world (in Barcelona, Spain; Tel Aviv, Israel; and Tokyo, for example) are made clearly visible on digital maps put together by a company called Mobileye: Speeding cars turn roadways red, and groups of cyclists glow like flames along bike routes. It’s a top-down view of all the traffic problems that drivers, cyclists, and pedestrians encounter and complain about every day.
The maps aren’t being used by city planners—at least, not yet. Instead, they’re part of Mobileye’s unique pitch to investors, journalists, and potential customers. The company, a subsidiary of Intel, can compile those red lines and glowing bike lanes because it has access to a network of millions of sensors built into privately owned cars around the world that share data from commuters whether they realize it or not.
If you drive a newer car, it’s likely to have at least one built-in camera or sensor that powers important safety systems such as automatic emergency braking (AEB) and blind spot warning (BSW), or that makes driving easier with assistance features such as adaptive cruise control and lane centering. Most of the software and algorithms that control those systems were developed by Mobileye.
But those same cameras and sensors might also be watching for potholes on the road and pedestrians on the sidewalk, then sending that information over a cellular connection you might not know your car had to a company you’ve probably never heard of so that it can create detailed data profiles of every roadway in the world.
Many Americans may be stuck at home right now due to the coronavirus pandemic, but when we all return to our normal driving lives, Mobileye’s technology will come with us. Every time you drive to work, take your kids to school, or go shopping, your car may be fitting another tile into the mosaic of a detailed composite map that can be sold in the growing market for car data.
“This type of data has enormous potential for public good, provided its collection is limited and handled in a way that protects drivers’ privacy,” says David Friedman, vice president of advocacy at Consumer Reports. “The problem is that currently there are no federal laws limiting the collection and use of that data, or even requiring clear disclosure of what is being shared, and with whom. It’s the Wild West out there.”
So far, Mobileye has mapped more than 180 million miles of roads worldwide—including the street in Brooklyn and the off-ramp on the New Jersey Turnpike—and it’s scaling up. As soon as this year, the company plans to use the sensors built into vehicles from BMW to collect data about their immediate surroundings—a process Mobileye calls “harvesting.” Similar sensors are built into vehicles from Ford, Nissan, and Volkswagen, and Mobileye says it will be harvesting data from them in the U.S. by 2021.
And Mobileye isn’t alone. A startup called Carmera that currently collects roadway data from commercial vehicles eventually plans to do the same with the sensors that are already built into production vehicles from Toyota. Tesla says it’s improving its partial autonomy software by using video data it collects from private vehicles, and other companies are close behind.
Sensors are also crucial to the development of the autonomous cars of the future, which will require highly detailed, up-to-the-minute maps to help them “see” the world around them. But these maps may also end up influencing city planning, development, and policing decisions where you live, work, and travel. How that data is used will depend on who gets their hands on it, and could lead to massive profits. Analysts say that data collected from cars could turn into a $750 billion industry over the next decade.
Carmera and Mobileye are targeting urban planners as potential users of these high-definition, or HD, maps. They say that cities will be able to use the data to discover where crashes take place and where drivers tend to speed, find out which buildings aren’t accessible to disabled people, identify roads where burned-out streetlights need to be replaced and potholes need to be fixed, and see where cyclists and pedestrians are concentrated so that infrastructure can be built to keep them safe.
But privacy and data collection experts tell CR that insurers could use data gleaned from HD maps to raise rates for drivers who travel on streets where other drivers tend to exceed the speed limit, landlords could choose to raise rents in neighborhoods with lots of luxury cars, credit reporting firms could make inferences about people who live on streets with less lighting or more potholes, and law enforcement agencies could target pedestrians, homeless encampments, or public gatherings.
Ben Green, author of “The Smart Enough City” (MIT Press, 2019) and a researcher at the Berkman Klein Center for Internet & Society at Harvard University in Cambridge, Mass., says the same data collected from cars to make cities “smarter” could have unintended consequences when used by other sectors, such as healthcare, human resources, and the media. “Maybe that data is then going to lead to profiles of you that then make an assessment when you apply for a job or apply for healthcare or any number of things,” Green says. “These systems can find ways, without breaking the law as it stands, of redlining people or excluding people.”
Raising a Car’s IQ
Today, most cars use cameras and radar sensors so that advanced driver assistance systems (ADAS) such as AEB can prevent potential crashes before they happen. As these systems get more complex and take over more vehicle functions from the driver, they are starting to double check what their sensors “see” against HD road maps that are accurate to within a few centimeters.
“HD maps are crucial as you start to get into higher-level automation,” such as when cars are able to automatically steer within lane lines, says Sam Abuelsamid, a mobility analyst at Navigant, a consulting firm. “If there’s snow on the road and you can’t see the lane, GPS is neither precise enough or reliable enough for that kind of application.”
In order for HD maps to work, they must be updated constantly—so ADAS systems are trained to spot differences between existing HD maps and the real world their sensors are seeing. If there has been a change—say, if temporary lines get painted because of road construction—the sensors can send an update to the HD mapmaker.
Mobileye is creating HD maps in a project called Road Experience Management, or REM. The company says that as few as 10 cars passing down a newly modified road could send back enough data to build an updated map. Outside the U.S., Nissan and Volkswagen are already using REM data to improve their driver assistance programs. And a spokesperson for the map provider Here Technologies told CR it would debut HD maps in BMW and Mercedes-Benz vehicles next year.
Tal Babaioff, who heads up the REM program at Mobileye, told CR that the data collected from cars is not only anonymized but also extremely pared-down. “We’re not sending images,” he said. “We’re not sending license plates. We’re sending semantic information about the world.” Neither images nor the data derived from them are stored on the vehicle.
Even without HD maps, municipalities can access an unprecedented amount of information about where and when people are traveling. For example, 70 cities—including Los Angeles and Austin, Texas—collect data from micromobility services such as e-scooters and shared bikes. Boston and other cities have installed sensors on municipal fleet vehicles to measure things like air quality, tree canopy coverage, and street lighting. Law enforcement agencies already use surveillance cameras, license-plate readers, and facial recognition software. And a massive trove of data about pedestrian and vehicle movement has already been collected from smartphone apps that share a user’s location—not to mention information shared from video doorbells, smart speakers, insurance companies, and fitness trackers.
The advent of Mobileye’s REM project, however, marks the first time that this kind of data is being collected from private vehicles, says Navigant’s Abuelsamid—but he predicts it won’t be the last.
“There’s others out there that are doing similar things, but definitely Mobileye is the first one to do this at scale with consumer vehicles,” he told CR. The company is the world’s largest supplier of ADAS sensor software, and the massive number of vehicles containing Mobileye software is a huge advantage for data collection. “They’re going to have the sensors in the cars anyway to power their ADAS systems, and they need the maps,” Abuelsamid says. (Navigant does not disclose its clients but does work with companies that sell or provide access to data collected from vehicles.)
Big Business
For companies such as Carmera and Mobileye, the depth of data collected by their sensors opens up numerous potential new revenue streams. That’s because HD maps aren’t just useful for improving automotive safety and self-driving systems. They can also be used to create inventories of electrical boxes and poles for utility companies, find potholes or faded lane markings, or alert cities to where new bike lanes or crosswalks should be built—what Carmera CEO Ro Gupta called a byproduct of HD map creation.
For example, the company uncovered an unexpected source of pedestrian data when it took the privacy-protecting step of blurring the faces of bystanders its cameras captured. “We realized, ‘Hey, we’re counting a person each time,’” Gupta says. “We could create this very granular block-level indexing score across the city of how dense any block in the city is on a given month, day of the week, or time of day.”
Gupta says that Carmera has abandoned its early plans to sell that data to focus instead on HD maps for automotive customers. Now, Carmera gives sensors to fleet operators so that they can monitor their drivers in real time in exchange for unlimited access to the data those sensors collect. It also shares some of that data with cities. (The company does not share which fleet operators it works with.)
Mobileye talks about utility companies, city planners, public transit agencies, and law enforcement as potential customers for its HD map data. “There’s absolutely a significant revenue source,” Abuelsamid says. “It’s definitely a new business for Mobileye. They see it as a new revenue stream.”
The company says its maps can cost a city or another third party a yearly fee in the hundreds of thousands of dollars, depending on how detailed they are and how much distance they cover. Babaioff says that in addition to safety improvements, cities would no longer have to conduct manual traffic counts of cars, pedestrians, and cyclists, which could make REM more cost-effective than traditional methods of collecting similar data. The company has already made a deal with a major U.S. city that has yet to be named.
These types of business arrangements are a huge business opportunity for the companies that have access to vehicle data: A 2016 white paper from industry research and consulting firm McKinsey projects that by 2030, data collected from vehicles may represent a $450 billion to $750 billion industry.
Privacy Concerns
How much control drivers have over the information their cars share largely depends on the privacy policies of the companies collecting it and—for now—the technical limitations of the equipment involved, including cameras, processors, and cellular networks.
Babaioff says that REM-equipped cars are intentionally designed to protect a driver’s privacy. “No images leave the camera,” he told CR. Data points are stripped of any identifiable information, and REM collects only segments of individual drives. “We will throw away the start and end of the drive,” Babaioff says. “If you drove from home to work, I would actually get [multiple different] segments that aren’t close to your home and not close to your work. This gives us ability to collect the data without you sacrificing any of your privacy.”
According to Mobileye, each vehicle broadcasts only about 16 kilobytes of road data per mile—about as much data as the text of this story. Overall, every car that harvests for REM will send out around 200 megabytes of data per year. That limitation isn’t only due to privacy concerns. It’s also about minimizing the cost and effort of storing and analyzing data from tens of millions of vehicles, then sending it back as updated HD map data. “That would be too much data, going both ways—both transmitting to Mobileye and transmitting it back to the vehicle,” Abuelsamid says. Even with such a small amount of data being sent from each REM-equipped car, Mobileye is able to harvest over 1.2 million miles of road data every day.
Alejandro Vukotich, senior vice president of the BMW Group’s driver assistance and autonomous driving development, said that BMW would not want raw camera data from the sensors in privately owned vehicles. “We are not interested in that,” he said. “It makes no sense, and no one would pay for it.” For example, a full hour of high-quality video can require over a gigabyte of storage.
Carmera, however, does broadcast video clips from its small fleet of partner vehicles so that humans can verify that its object-identification algorithms are working properly. Gupta says it would eventually be possible to use data from cameras installed in privately owned Toyota vehicles, but because of privacy laws—which he says are “good things”—and technical hurdles, “it’s going to take a long time for that to be available en masse,” he told CR. (The Toyota Research Institute, which has partnered with Carmera, did not respond to CR’s inquiries about the project.)
Even Tesla—which isn’t planning to rely on HD maps and instead sends what it calls “short video clips” from built-in cameras in its vehicles back to the company to improve its Autopilot driver assistance software—can’t collect that much data, Abuelsamid says.
Although the company has publicly released data from an owner’s car after a crash, a Tesla vehicle doesn’t save or broadcast everything it sees. “It’s selectively picking small tidbits of data and saving that because there just isn’t enough storage space on the vehicle to store that data, and it would be prohibitively expensive to transmit it all back to Tesla.”
Drivers can opt out from harvesting REM data, though Mobileye says how they can do so depends on rules set by the car manufacturer. BMW says that drivers automatically opt in to REM when they activate the car’s connected services and that they may lose certain connectivity features if they opt out, such as warnings about slippery roads and traffic signs. (Nissan and Ford didn’t respond to CR’s questions about REM, Tesla didn’t respond to requests for comment, and a spokesman for Volkswagen told CR the company doesn’t “go into detail on specific technology suppliers and their specific hardware and software solutions.”)
But if how automakers currently communicate about privacy is any guide, drivers might not even realize they’ve opted in, says Johanna Zmud, a senior research scientist at the Texas A&M Transportation Institute in College Station who has studied automotive data privacy.
Plenty of carmakers already collect some data from private vehicles, Zmud says, usually with a privacy agreement that says they need the information “for diagnostic purposes.” In most cases, these policies may flash across the screen for a few seconds when a car is started, or it might be included in the stack of documents buyers have to sign at the dealership, buried between odometer disclosures and finance applications.
“I’ve seen some of the agreements that, for instance, automakers have for a purchaser and they’re very, very long,” Zmud told CR. “Most people just go to the last page and sign.”
Even drivers who say they don’t want their cars to share data may still allow their cars to do so, depending on how difficult it is to opt out, and whether opting out disables certain features. “There’s this concept called the privacy paradox,” Zmud says. “People say that they care about keeping their data private, but their actions belie that. They don’t act accordingly.”
The ability for a company to sell data collected from private vehicles may become more difficult in the future, according to Dorothy Glancy, a professor at the Santa Clara University School of Law in California who focuses on privacy and transportation. New laws in Europe and California that seek to better define control over personal data may threaten the revenue model that Mobileye and others rely on. “That issue will be very contentious over the next couple of decades,” she said.
Mobileye is aware of how this controversy may affect its REM project. In a 2016 filing with the Securities and Exchange Commission, the company listed future legal concerns about data privacy as a potential harm to its business. “It is possible that these laws may be interpreted and applied in a manner that may be inconsistent with our data practices,” the filing said.
Societal Impacts
Lorrie Cranor, director of the CyLab Security and Privacy Institute at Carnegie Mellon University in Pittsburgh, told CR that even if it seems like companies like Mobileye are handling the data they collect responsibly, that doesn’t answer the question of what other groups might do with it once it is shared or sold.
“If the data is going into algorithms to figure out where we need new traffic lights, and it’s never going to be released as a dataset, and deleted after a certain amount of time, it seems like it’s relatively low risk,” she says. That’s especially true when compared with more revealing datasets, such as information collected from mobile phone apps and traffic cameras.
But, Cranor says there’s no guarantee Mobileye or others in a similar business won’t sell data to third parties that may use it for purposes other than traffic planning.
Who gets to use large datasets largely determines how they get used, says Vinhcent Le, legal counsel at the Greenlining Institute, an advocacy group that aims to advance economic opportunities for people of color. Le said that he sees great benefits for cities to use data for good—but in his experience, data that’s sold usually ends up in the hands of those who have the most money or power.
“The potential is there for city planners, but in my head what happens more often than not is that private corporations, developers, people who are trying to sell you something are the ones who end up with that data, and they find ways to monetize it, and that ends up harming the same folks we’re supposed to be helping,” he told CR.
Consider Amazon’s rollout of same-day delivery in 2016. Although Amazon didn’t include race as a factor when choosing which neighborhoods to debut same-day delivery, an investigation by Bloomberg found that the service initially excluded predominantly black neighborhoods in several major cities. “What they ended up doing is denying people services on the basis of race even if they didn’t use race,” Le said.
Similarly, real estate developers or analysts could use the data to identify neighborhoods where luxury cars are parked, or how many grand opening signs are going up on a block—all ideas initially proposed by Carmera in 2017 that Gupta said the company has since abandoned. At the time, Carmera said this information could be used by real estate brokers, architects, insurers, zoning officials, and industry analysts.
It can be difficult to restrict how private companies use the data they collect, including HD maps. That’s due to an absence of regulation, says Justin Brookman, CR’s director of consumer privacy and technology policy.
“Real-time data collection can do a tremendous amount to promote safety and facilitate repairs, but people are right to worry about what their cars are sharing, and with whom,” he says. “The default is that companies can basically do whatever they want with data, absent some specific statute.”
Historically, Brookman says, companies have argued that privacy statutes violate their First Amendment rights, and courts have agreed. For example, in 2011 the Supreme Court struck down a Vermont law prohibiting pharmaceutical companies from accessing doctors’ prescribing records, saying that the law interfered with the companies’ ability to tailor their marketing messages.
And then there’s the question of law enforcement. While privacy advocates say that police investigations would be more likely to rely on cell-phone data, electronic tolls, traffic cameras, and a vehicle’s own event data recorders (which can capture how a car was being driven prior to a crash), police could still determine information about a driver’s daily patterns and “deanonymize” HD map data—especially if they already have access to those other databases.
They could also use HD mapping data to make decisions about how and where to police, which Le says may reinforce existing inequities. “We have all this mapping that shows you where people are jaywalking or where homeless people are,” he says. “That’s where you need to flip the script—that’s not where you need to send police, that’s where you need to send more services.”
Cities Get Smart
While it might be difficult to control what private companies do with data collected from vehicle sensors, there’s already precedent for controlling data usage at the municipal level. Cities such as San Francisco and Oakland, Calif., and Somerville and Brookline, Mass., have prohibited city agencies from using facial recognition data, and cities and counties across the country have banned automated license plate readers.
The Los Angeles Department of Transportation operates on a policy of what privacy experts call data minimization—collecting as little data as possible to minimize the chances it gets abused—when it monitors and manages micromobility services. “The system is built to process only the minimum amount of vehicle data needed to fulfill our responsibilities to the public,” said spokesman Colin Sweeney. In addition, the Los Angeles City Council formally adopted data protection principles developed by the LADOT.
Kris Carter, co-chair of the Mayor’s Office of New Urban Mechanics in Boston, says the city has already had to make decisions about whether to use data that may not adequately respect the privacy of its residents.
“There’s datasets you can buy out there that are generated by mobile advertising apps that people have on their phones that show pedestrian movements at a very fine degree,” he says. “I think for cities, there’s a fairly good question about whether we want to own and participate in that data collection of residents or not. And that’s still an open question, I think.”
For Carter, who has already collected data from sensors on municipal vehicles and smartphone apps developed specifically for use by Boston residents, a city must be transparent in how it chooses to use data it collects on its residents. “We want to make sure that people are aware of how government is using those types of datasets.”
One way of doing that, Le says, is by directly involving residents when cities choose whether to use new datasets that are available to them. “The solution is really democratic participation,” he says. “The people who live in that city should have some say in the technologies that are going to be used in their cities.”
But even if large datasets are being used responsibly by public entities, it doesn’t control what happens in the private sector. The public must also understand that the dystopian idea of a surveillance state isn’t necessarily a product of government, says the Berkman Klein Center’s Ben Green.
“We have a government-centric notion of surveillance and privacy,” he told CR. “In terms of Big Brother, we don’t want the government knowing what we’re doing and police knocking down the door. But we also need to think about surveillance from private companies—how does that lend them incredible amounts of power, not just over us as individuals but also over the public sector?”
Brookman says that ultimately it’s up to lawmakers to develop provisions that protect people against the misuse of data collected by this new technology.
“We shouldn’t have to trust companies to do the right thing, since their first obligation is to their shareholders, not drivers,” he says. “Americans deserve some basic level of privacy law to ensure that data isn’t being collected and shared in ways that are contrary to our interests,” he adds. “Right now, all bets are off.”
