FRAGMENT

Documentation

N Export Data#

a. S3 Export

#

FRAGMENT supports exporting your Ledger data to AWS S3. This makes your data available for analytics, regulatory compliance and ingestion into third-party systems.

Export format#

Data exports are delivered approximately every 5 minutes. Each data export contains the Ledger data created or updated since the previous export.

Exports are divided by data type into three files. Each file contains newline-separated JSON, where each line represents an individual instance of the respective data type: 'LedgerEntry', 'LedgerLine' or 'LedgerAccount'.

The files use a naming scheme of {File Prefix}/{type}/day={day}/hour={hour}/{partition}.part.

Example files:

Onboarding#

To enable data export, first create an S3 bucket. No special settings are required, but take note of the bucket name and AWS region.

Once the bucket exists, navigate to the Settings -> S3 Exports subsection of the FRAGMENT dashboard, and follow the instructions to create a new export. You need to provide:

  • S3 Bucket Name, the name of the S3 bucket in AWS.
  • S3 Bucket Region, the AWS region of the bucket created in the previous step (for example, us-east-1).
  • Export Name, a name for this export. This is for display purposes only.
  • File Prefix (optional), the path where exports are stored. If no file prefix is provided, exports are stored in the bucket root.

The instructions in the dashboard include applying a S3 bucket policy to your bucket. Depending on how you manage your infrastructure, this may take some time, so the onboarding flow in the dashboard can be restarted at any time.

Testing#

Once you've onboarded your data export, the dashboard lets you test the connection by writing a sample file to test/testLedgerEntry.part in your bucket. The test result can be:

  • Policy Not Applied, FRAGMENT received an authorization error. This typically means the provided resource policy has not yet been applied within your AWS account. This may also mean the provided bucket name or region differs from the one created in the initial setup.
  • Invalid Bucket Name, the provided bucket name does not exist in AWS.
  • Incorrect AWS Region, the provided bucket exists in a different region than provided.
  • Verified Permissions, the file was successfully written.

FRAGMENT does not remove the test file after verifying its existence.

b. Retool

#

You can add FRAGMENT as a GraphQL resource in Retool.

  1. Create an API client in the FRAGMENT dashboard.
  2. Add a new GraphQL Resource in Retool.
  3. Set the Base URL to your API URL.
  4. Add the Authorization Header with the value Bearer OAUTH2_TOKEN. Retool will replace OAUTH2_TOKEN with the actual token at runtime.
  5. Set Authentication to OAuth 2.0
  6. Enable Use Client Credentials Flow
  7. Set the following values:
    • Set Access Token URL to OAuth URL
    • Set the Client ID and Secret
    • Set Scopes to OAuth Scope
    • Set Prompt to consent
  8. Save the resource.

Note: Testing the connection in Retool may fail even when the resource is configured correctly. To check, use the resource in an app by running the query.