By: Casey Miller
I recently mapped ISP providers across the country so we could have a data-driven view of the Net Neutrality debate. Users can search an address or click an area on the map to see exactly how many ISP options they have. To do this, I made a custom base map using Mapbox Studio, generated a tileset from the massive GeoJSON files for each state using Tippecanoe, and I built-in search logic that incorporates our geocoder. I’ll walk you through how I built it.
Preparing the data
The first step was to download the Internet service provider data from the FCC, which is in CSV format. The file contains 68.4 million rows, one line per provider and census block pairing, and weighs in at 10.65 GB. At first, I considered keeping the additional provider information associated with each entry in the new data I was creating, but I soon realized that this made the size of the file too unwieldy. Instead, I opted to just maintain a count of the providers per block that would determine the color of the census block on the map.
In order to generate the data I needed, I wrote a python script that counts the providers in each census block, filtering out satellite provider entries and entries where the ISP doesn’t provide consumer service (some ISPs only provide business service).
Generating a tileset with Tippecanoe
Next, I downloaded the census block shapefiles from the U.S. Census Bureau. The Census Bureau provides the census block data as shapefiles by state, so I ended up with 51 (don’t forget D.C.) separate shapefiles to work with. Since the census block data is so granular, I decided to first go ahead and simplify the shapefiles a bit — enough that it wouldn’t really be noticeable to the user, but would save gigabytes of space and hours of processing time. While there are many ways to do this, I chose to write a python script to handle this for me. The script opens the directory that all of my shapefiles are in and simplifies them a tiny bit using ogr2ogr.
Now that I had my FCC data and my census block shapefiles in usable formats, there were a couple of different routes that I could have taken to map the data. That said, I chose to convert my shapefiles to GeoJSON so that I could add the provider counts as a property to each census block. Then, later, I could use the provider property value to set the color of the census block.
Script to convert shapefiles into GeoJSON:
Script to add PROVIDER property to each census block:
Now I have a folder containing 51 GeoJSON files (again, one per state) that also includes the provider count associated with each census block. Rather than combining these 51 files into one giant GeoJSON file, I opted to skip this step and let Tippecanoe handle this for me.
Tippecanoe creates tilesets out of collections of GeoJSON features, which made it perfect for this project. Trying to load 10.27 GB of GeoJSON data on a map would have been, well, impossible. Tippecanoe to the rescue.
The command I used outputs a tileset with a minimum zoom of 14, detects and combines shared polygon borders, merges all of the GeoJSON layers into one, and only includes the PROVIDERS property, dropping all others.
Command to run from inside the folder with GeoJSON files:
tippecanoe -o census_blocks.mbtiles -Z14 -z14 -ab -l --include="PROVIDERS" *.geojson
After generating my tileset with Tippecanoe, I uploaded the output as a tileset using the Studio editor.
Then I added it as a layer to my map style.
And styled the layer according to the PROVIDER property that I included for each census block in my GeoJSON.
Now I have a basemap with a census block overlay styled according to the number of providers for each census block.
Creating the search logic and adding the geocoder
Rather than storing all of the detailed provider data for 11 million census blocks, the app queries two FCC APIS to return the relevant ISPs for the current location. First, the app obtains the latitude and longitude of the current specified location one of three ways:
- If the user allows the app to use their current location, the latitude and longitude is sourced from the response provided by the browser’s geolocator
- If the user enters an address in the search box, the Mapbox geocoder returns the associated latitude an longitude
- If the user clicks a location on the map, the latitude and longitude are pulled from the click event
After obtaining the current coordinates, the app queries the FCC’s census block API. The latitude and longitude are fed in as parameters, and the response includes the census block FIPS code for those coordinates.
Here’s an example request and response:
After parsing the FIPS code from the response above, the app queries the FCC Fixed Broadband Deployment Data API to get the providers currently serving that zip code.
While the API returns all providers, the app filters out any providers that only offer satellite service, as well as those that don’t provide consumer service (just like we did when aggregating the provider counts above before adding them to the GeoJSON).
Keep building
Here’s the full map. You could add a way to zoom out farther on the map in order to get a regional, or possibly even countrywide view of the provider diversity, or lack thereof. Additionally, it might be nice to include the speed offering associated with each listing.
Looking for more inspiration? Check out Santa Rosa fire map: How I built it; MSF Search and Rescue map: How I built it; and How I mapped the leaked changes to national monuments. Explore the tutorials on our help page to learn more about how our tools fit together.
The Net Neutrality map: How I built it was originally published in Points of interest on Medium, where people are continuing the conversation by highlighting and responding to this story.