Setting up Cargo on Google BigQuery


Cover image

In this tutorial, we will guide you through setting up your Google BigQuery as a store of records in Cargo. This setup ensures Cargo has the necessary permissions in Google Cloud to read and write data efficiently.


Overview

Permissions

To ensure the safety of your data, Cargo has a few limitations on what it can and cannot do inside your Google BigQuery instance.

What Cargo can do:

  • Read data from schemas and tables, even if they are spread across multiple databases
  • Write data into new schemas and tables

What Cargo will never do:

  • Overwrite existing schemas and tables (Cargo always creates its own schemas and tables when needed)
Step 1

Enable the necessary APIs

Cargo uses some Google APIs that must be enabled. To do so:

  • Go to the Google Cloud Console.

  • Select APIs & Services.

  • Select Enabled APIs & Services.

  • Search for and enable the following APIs: BigQuery API, Cloud Resource Manager API

Step 2

Create a storage bucket

To enable Cargo to load and unload data from BigQuery, we need access to a dedicated storage bucket for this purpose.

To create a new bucket:

  • Go to the Google Cloud Console.
  • Search Object storage in the search bar.
  • Create a new bucket and follow the steps.
Step 3

Create a Service Account

Cargo will use this service account to access the APIs enabled in the previous step. To create a service account, follow these steps:

  • Go to the Google Cloud Console.
  • Click on IAM & Admin.
  • Click on IAM.
  • Click on Service Accounts.
  • Click on Create service account.
  • Give the service account a name.
  • Grant the following roles:
    • BigQuery Data Editor
    • BigQuery Job User
    • Storage Object User
  • Click on Done.
Step 4

Generate a Key

Follow the steps below to generate a key:

  • In Service Accounts, click on the created service account.
  • Click on Keys.
  • Click on Add Key.
  • Choose Create new key.
  • Select JSON.
  • Click on Create.
  • This should download a file to your computer.
Step 5

Creating a New Workspace on Cargo

To create a new workspace on Cargo, follow these steps:

  • In the Create new workspace form, define a prefix that Cargo will use to prefix the datasets and schemas it creates, in order to avoid any conflicts.
  • Copy and paste the content of the file into the field labeled Service Account.
  • To determine which location to use, follow these steps:
    • In the Google Cloud Console, open BigQuery.
    • Choose a dataset.
    • Look for the Data location value.
Outcome

Setup completed

You are now ready to use Cargo with Google BigQuery!