Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Bolo.

Table of Contents

Search

  1. Introducing Mass Ingestion
  2. Getting Started with Mass Ingestion
  3. Connectors and Connections
  4. Mass Ingestion Applications
  5. Mass Ingestion Databases
  6. Mass Ingestion Files
  7. Mass Ingestion Streaming
  8. Monitoring Mass Ingestion Jobs
  9. Asset Management
  10. Troubleshooting

Mass Ingestion

Mass Ingestion

Google BigQuery targets

Google BigQuery targets

The following list identifies considerations for preparing and using Google BigQuery targets:
  • Download and install the Google BigQuery JDBC driver.
    1. Download the Google BigQuery JDBC driver zip from the Google Cloud website.
    2. Copy all of the jar files in the installation zip to the following directory:
      Secure Agent installation directory
      /apps/Database_Ingestion/ext/
    3. Restart the Secure Agent.
  • Ensure that you have a service account in your Google account to access Google BigQuery and Google Cloud Storage.
  • Ensure that you have the client_email, project_id, private_key, private_key_id, client_id, and region ID values for the service account. You must enter these details when you create a Google BigQuery connection.
    Specify the private_key_id and client_id values in the
    Provide Optional Properties
    field of the connection properties. Use the following format:
    "private_key_id": "<
    private_key_id_value
    >", "client_id": "<
    client_id_value
    >"
  • If you want to configure a timeout interval for a Google BigQuery connection, specify the timeout interval property in the
    Provide Optional Properties
    field of the connection properties. Use the following format:
    "timeout": "<
    timeout_interval_in_seconds
    >"
  • Verify that you have read and write access to the following entities:
    • Google BigQuery datasets that contains the target tables
    • Google Cloud Storage path where Mass Ingestion Databases creates the staging file
  • To write data to a Google BigQuery table, you must have the following permissions:
    • bigquery.datasets.get
    • bigquery.datasets.getIamPolicy
    • bigquery.models.*
    • bigquery.routines.*
    • bigquery.tables.create
    • bigquery.tables.delete
    • bigquery.tables.export
    • bigquery.tables.get
    • bigquery.tables.getData
    • bigquery.tables.list
    • bigquery.tables.update
    • bigquery.tables.updateData
    • bigquery.tables.updateTag
    • resourcemanager.projects.get
    • resourcemanager.projects.list
    • bigquery.jobs.create
  • Mass Ingestion Databases
    loads source data in bulk mode to Google BigQuery targets.
  • For database ingestion incremental load tasks, you must enable source database Change Data Capture (CDC) on all source columns.

0 COMMENTS

We’d like to hear from you!