Skybridge

This is a mirror of my senior thesis proposal for now. As more progress is made, I'll update it.

1. Background

The rise of the cloud has allowed for multi-device and multi-user applications and websites that would have been impossible beforehand. However, the fact your data is stored on another company's servers creates a data silo, where the only way to access your data is through that company's app/website. While regulations have made it so most apps have some form of accessing or exporting your data, the methods to do so are disparate--some have nice APIs, but many are more complicated, requiring logins and email requests to get data. Importantly, they're usually too complex to reliably allow developers to create new interfaces using that data.

2. Goals

For this thesis, I want to research and prototype a protocol to make it easier for developers to interact with cloud (and other 3rd party) data.

3. Potential solutions

To solve this 3rd-party sync problem, I want to explore two principles: local-first and composability.

Due to the disparity of 3rd-party sync solutions, there are often few reliable ways to constantly update data. One issue is timing: you can automatically export your Google data to your Drive, for example, with it's Takeout tool, but it can only be done once every two months. Another issue is overlap: Spotify lets you easily access your last 30 days of listening with an API, but to get a full history it requires reconciling with a zip file that takes a month to email to you.

This is where local-first principles come in: there are proven methods of reconciling data sources that might provide overlapping information or only sync every so often. This is especially important considering the obstacles that often occur during the sync process: if an API goes down, or a scraper runs into a CAPTCHA, it's helpful to have a redundant network of data syncing methods, which need to handle overlaps.

The other issue with this disparity is that it's currently a ton of work for developers to create all the necessary components for a 3rd-party integration. That's why it's crucial to have this protocol be an open source, composable ecosystem. For example, if a developer can utilize an already-built Google Drive integration (made by some other developer), it removes a huge barrier to interacting with those bimonthly takeouts.

4. Research questions

In order to develop the prototype, there are three main lines of questioning I want to research:

1. What are the ways 3rd-party apps expose data? What are the barriers to accessing it?
2. What does a great composable ecosystem of integrations look like? How should data be represented and stored?
3. Based on the "inputs" from Q1 and the "outputs" from Q2, how can we most easily create translations (the integrations themselves) between the two? What building blocks are most useful to help developers build these integrations?

5. Motivation

In the world before the cloud, users could choose which interfaces they used to view and edit their data. A user of a Web 1.0 application, like email, was able to choose any email client to interact with it. If they needed one that was accessible for a certain disability, or had great keyboard navigation, or they just liked the aesthetics, they could pick and even switch between those different interfaces.

With cloud apps, that choice is usually gone: to use Spotify, you're locked into Spotify's app, no matter how many UI changes they make. This is a problem for the cloud companies too: even if they want a focused and streamlined product, the lack of interface choice leads them to implement unneeded features to accommodate small segments of users.

By enabling developers to interact with cloud apps' data, we can reintroduce that choice for users while still keeping the benefits of the cloud and the apps we've come to rely on.