Precog: Uniform Access to Any API
By Chris Dima
April 28, 2020

The number of web APIs has grown exponentially over the past decade. Today, nearly every product in every sector exposes a public API, and organizations commonly deploy private APIs to share data internally. The industry has settled for ad hoc approaches to accessing API data, commonly using a combination of Python and command line tools. And so we tacitly assume that the only way to access API data is using custom code. Precog changes that.

To motivate Precog’s solution to the API problem, we need to understand where the true complexities in API access lie. We ask “Is there an API?”, and then feel satisfied when the answer is “Yes”. But the existence of an API does not mean that we can easily access the data in a meaningful way. This brings us to the three primary complexities of using web APIs: setting up the API requests, reshaping the data into an analytics-ready format, and making the reshaped data accessible.

Setting Up API Requests

Every API requires a different setup. Some APIs use basic authentication while others use OAuth 2.0. Many APIs are paginated, but pagination is not standardized: some APIs return the next page in the response header, while others return a pagination token or offset nested in the response data. Sometimes you have to make an initial request to gather some metadata, which is then used in subsequent requests. It’s also common to make requests from multiple endpoints.

Without Precog, accessing an API means cobbling together all of the above components in a custom script. But writing custom scripts is not a sustainable business solution, as it consumes developer resources that could otherwise be used to directly impact the business. Precog solves this problem by providing a uniform user interface to access API endpoints.

No matter the API, the interface looks the same. Precog authenticates using the authentication type of your choice. Precog makes multiple calls to the API to seamlessly handle pagination. The end result: Precog streams API data directly from the API to the data store of your choosing, allowing you to reshape it along the way. This brings us to the next complexity.

Making API Data Analytics-Ready

Our ultimate goal is to gain understanding of API data through analysis and visualization. But analysis and visualization tools speak the language of two-dimensional homogeneous data (like a spreadsheet), which is categorically opposed to the multi-dimensional heterogeneous structure of API data (think nested spreadsheets with missing entries). The industry currently solves this problem again with custom code. Data engineers are tasked with deciding the correct way to reshape the complex data into two dimensions while preserving the meaning of the row. Precog has solved this problem in the general case.

Nearly all APIs return JSON data, due to its flexibility and readability. But this comes at a hidden cost: the relationships described in a JSON document are deeply complex in unobvious ways. Furthermore, the structure of JSON responses varies wildly from API to API. Some APIs return an array of objects, where each object represents a semantic row. Others return one or more objects that encode metadata alongside the data. Others use array indices to encode semantically meaningful information. Also, the data are frequently heterogenous, meaning that each nested array or object looks different than its neighbors. Precog innately understands unstructured data. No matter how deeply nested. No matter how many holes. Precog understands API data.

Precog’s data browser presents a uniform view of all API data, enabling you to immediately see your data in a browsable format and to pick the components that are relevant to you. Precog then transforms your data into the two-dimensional format that data analytics tools understand, while preserving the semantic relationships encoded in the JSON structure. In this way, Precog empowers everyone to transform complex API data into tabular, analytics-ready data.

Loading API Data into a Destination

The third complexity of API access is loading the reshaped data into a destination. Each destination provides a different way to load data, often requiring an engineer’s expertise. Precog connects to your database, pushing in the reshaped API data with the click of a button. With Precog, there is no need to maintain your database in parallel with your loading tool.

Precog provides a single solution for streaming API data from the source, transforming it into analytics-ready tables, and loading it into your target destination. Precog gives everyone direct and instant access to any API.

Web API access remained an unsolved problem until Precog.

Download free trial here. Buy it here.


Ready to Start?



We chose to use Precog because they were the only company willing to handle our complex data connections. Precog was extremely helpful getting us set up and running smoothly. Since then it has been one of those tools that just works solidly and reliably which is one less thing our team nee... Read More

Derek Binkley - Engineering Manager

Precog is an important partner for Cured and a critical member of our data stack. The Precog platform has delivered data connectors to necessary data sources other vendors could not or would not, and in a very short timeframe. The product is intuitive, efficient, cost-effective, and doesn&... Read More

Ashmer Aslam - CEO Cured
Walnut St. Labs

Precog lets us prototype analytics projects quickly — building marketing dashboards based on data from a variety of sources — without needing a data engineer or developer — we create new data sources in a few hours to sources like Brightlocal, a popular local SEO SaaS solution, and h... Read More

Chris Dima - CEO

We welcome Precog to the Alteryx technology partner ecosystem as a partner extending the capabilities of our platform, further simplifying analytics for our customers.

Hakan Soderbom - Director of Technology Alliances

We recognized a need in our customer base to perform advanced analytics on SAP data sets — we performed an extensive evaluation of Precog and chose it as a strategic solution for our go to market needs based on its performance and given their strong strategic relationship with SAP.

Alfredo Poncio - CEO

Precog is the vital tool in our ability to pull data from a variety of business sources quickly and cleanly. Our internal MongoDB backend, as well as other cloud services like Hubspot, were a constant challenge to the business teams desire for reporting data prior to using Precog. With the... Read More

Josh Wilsie - VP