Cost-effective scaling with distributed and parallelized processing
By: Young Hahn
New Mapbox blog posts are now published on www.mapbox.com/blog. Leave your email to receive email updates when new posts are published or subscribe to the new RSS feed.
Mapbox Tiling Service (MTS) allows the processing of massive sets of data into custom vector tilesets, continuously updating the maps as the data changes. MTS is now available in public beta to all Mapbox developers as a service for uploading source data with transformation rules called recipes.
Tilesets generated using MTS can then be styled in Studio and composited with any other vector tilesets.
Whether mapping 5G coverage, determining the effectiveness of an ad campaign per zip code, analyzing new user signups per country, or visualizing local events in a social app, or tracking the spread of COVID, MTS is a service that integrates custom datasets of any scale into maps faster, cheaper, and with more flexibility and control than previously possible. MTS allows developers to focus on app development, not mapping infrastructure.
Scale and Speed
We first built MTS internally, for us to process daily +300 million miles of anonymized and aggregated live road telemetry data and live update our maps. We designed the services to not only process big data that was continuously changing, but to transform this data into maps that served out to more than 650 million monthly active users — to support the scale of huge customers such as Facebook, Snap, the Weather Channel, Tableau, and Shopify.
Yahoo! Japan uses MTS to create time series visualizations of live weather in Japan that update every 5 minutes. Yahoo! Japan recently migrated all of their weather maps to MTS so that they could focus on improving the core experience of their apps and stop thinking about maintaining mapping infrastructure, and in doing so their team was able to save engineering time and money while providing a better experience for their users.
“By migrating all of our minutely updating weather maps to MTS, we were able to save engineering time and money while providing a better experience for our users. We’re excited to migrate more of our maps to MTS.”
— Soichi Takahashi, Unit Manager at Yahoo! Japan
Distributed and parallelized processing
MTS creates and updates data using distributed and parallelized processing, meaning data is processed much more quickly than is possible with a standard, single server setup with comparable tools. For example, a global basemap at 30cm precision can be processed in under 2 hours with MTS, whereas a comparable workload would take multiple days to process on a single server.
Multilayer tilesets
Map styles are more performant if they reference fewer tilesets. MTS has the flexibility of organizing data into multiple vector layers that are still ultimately part of the same tileset. This allows more data on the map, without sacrificing the same data organization benefit of having multiple tilesets.
Snap uses MTS to power their Places map in Snap Map, letting users discover local business and see rich metadata about each venue, like ratings and pictures. MTS allowed Snap to remove operational complexities and cut their total data processing time from 4 hours to just 15 minutes.
Recipes: Increased customization to build compelling experiences
With MTS, it was important to choose an architecture that would deliver unprecedented processing performance while putting control directly in developers hands for optimizing every aspect of the map, from the performance to the visual design. This is why we created recipes, JSON documents composed of configuration options that tell MTS how to process data into vector tiles. Recipes provide options for generating vector tiles such as degree of simplification, zoom level extent, geometry unioning, attribute manipulation, etc, see the detail about all the available configuration options in the Recipe reference.
Recipes specify the precision of the data, filter features, and manipulate data using SQL-like statements and Mapbox’s expression syntax. This provides easy access to cartographic operations such as unioning, aggregation, label pre-collision, and ranking in an easy to use form factor to improve the legibility, performance and design of the map data. Similar functionality can be built using utility functions in PostGIS but does not have the ability to scale.
AllTrails uses MTS to map one of the largest databases of custom hiking trails available for offline users via our SDK.
“MTS allows AllTrails to quickly upload and process thousands of custom trails for our users. It makes it extremely easy to provide our customers with up-to-date information of highly detailed trail maps for their hiking, biking, and camping adventures around the world!”
— James Graham, Head of Engineering at AllTrails
Regionally redundant and auto-scaling caching network
MTS runs on Mapbox’s global platform, allowing data to be immediately available across multiple regions where Mapbox operates and is automatically backed by our regionally redundant, auto-scaling caching network, making it seamless to scale, in any geography, with low latency and high throughput.
Multiple regions with dynamic routing makes Mapbox’s infrastructure resilient to single-datacenter or single-region events like natural disasters and internet infrastructure outages.
Process data with MTS once, and leverage that data in many different formats. Once data is processed by Mapbox, it is available in vector tile for on-device rendering of maps, in addition to other form factors like static maps, map thumbnails, overview images, legacy GIS clients, and new interfaces (e.g. iOS14 Widgets).
This provides enterprise-grade data processing infrastructure and reliability without needing to spend the time and money to set up, build, deploy, or maintain it.
With MTS we’ve abstracted away the hard stuff without taking away any control. Plume Labs, uses MTS to power their geo-datastore, which monitors global air quality.
“MTS helps Plume Labs get people from A to B in the healthiest way possible. Using MTS, we’re able to update the entire world’s air pollution data in real-time so that our users can avoid pollution hotspots while they walk, run, or cycle through their city.”
— Boris Quennehen, Atmospheric Scientist at Plume Labs
With MTS, instead of managing a set of servers, developers stream data to Mapbox and get enterprise-grade tilesets available to all users on any device within minutes.
Private Data
Maps are locked and set to private by default, giving developers total control over who is looking at data. Mapbox never accesses customer created data layers. Security, managed with access tokens, lets developers control permissions: our management interface allows developers to create, revoke, and monitor tokens. For more information, see our Tokens API documentation. SAML single sign-on is also available for all Mapbox customers, enabling teams with multiple users to collaborate more securely on the platform.
Get started in minutes
To start using MTS we’ve created a library of recipe examples packaging the learnings and best practices from early MTS customers like Yahoo! Japan, Snap, AllTrails, T-Mobile, and Plume Labs.
Browse the use case-specific examples and immediately drop them into projects, or customize them. Start with MTS by using our handy CLI utility.
Introducing Mapbox Tiling Service was originally published in maps for developers on Medium, where people are continuing the conversation by highlighting and responding to this story.