Integrations API Overview
Integrations in Fibery are quite unusual. It replicates a part of an external app domain and feed data into Fibery and create several Databases.
Dedicated service (integration application) should be implemented to configure and fetch data from an external application.
All communication between integration application and Fibery services is done via standard hypertext protocols, whether it can be HTTP or HTTPS. All integration applications are expected to adhere to a particular API format as outlined in this documentation. The underlying technologies used to develop these integration applications are up to the individual developer.
Users may register their applications with Fibery by providing a HTTP or HTTPS URI for their service. The service must be accessible from the internet in order for the applications gallery to successfully communicate with the service. It is highly recommended that service providers consider utilizing HTTPS for all endpoints and limit access to these services only to IP addresses known to be from Fibery.
In essence, Fibery's applications gallery service acts as a proxy between other Fibery services and the third party provider with some added logic for account and filter storage and validation.
Installed application will be available for all users in your Fibery account. Users from another accounts won't be able to see or use your integration application.
How to add, edit or delete
Let's assume you created an integration app, made it's url available and are ready to try it in Fibery.
Adding custom app
Navigate to space you would like to integrate and click integrate button, find add custom app option and click. Follow the flow.
Editing custom app
You can change the link to a custom app or force an update to the configuration of the app after deploying it to production by finding your app in catalog and clicking on settings icon in right upper corner.
NOTE: It is recommended to update custom app every time when changes to config or schema of app are happened. Just click update button in described case.
Deleting custom app
You can delete a custom app by finding your app in the catalog and clicking on settings icon in right upper corner.
NOTE: In case of app deletion, the databases will not be removed and relations to the databases will not be affected.
Domain
App schema
Each app should follow strict schema with the following structure (all fields are required):
Name | Type | Description | Example |
---|---|---|---|
name | string | Name | "MyApp" |
website | string | Website | "http://myawesomeapp.com" |
version | string | Version | "1.0.0" |
description | string | Description | "My awesome app" |
authentication | Array | Authentications | [{id: "none", name: "no auth"}] |
sources | Array | Empty array | [] |
responsibleFor | Object | Responsibilities | {dataSynchronization: true} |
Authentication
Authentication model represents type of authentication in app and has following structure:
Name | Type | Description | Required | Example |
---|---|---|---|---|
id | string | Identity of authenticaiton | true | "oauth" or "token" |
name | string | Name of authentication | true | "Connection Settings" |
description | string | Description of authentication | false | "Give the name for connection" |
fields | Array | Authentication fields | false | [{optional: false, id: "accountName", name: "Title", type: "text", description: 'Give the name for connection'}] |
If your app doesn't require authentication you should add authentication with id = "none". In the case you can omit fields.
Authentication field
Authentication field represents the field in account. So account that will send to API endpoints will consist of the authentication fields. Authentication field has following structure:
Name | Type | Description | Required | Example |
---|---|---|---|---|
id | string | Id of field. The id will be specified in account object. | true | "accountName" |
name | string | Name of the field. Will be displayed in auth form. | true | "Title" |
type | string | Type of the field | true | "text" |
description | string | Description of the field | true | "Give the name for connection" |
optional | boolean | Is the field optional | false | false |
value | string | Default value of the field | false | null |
Read about field types. Note that authentication field with some types requires more information to specify. For example, for highlightText type you also need to specify editorMode in authentication field.
Filter
Filter represents filter that will be applied on data. Filter has following structure:
Name | Type | Description | Required | Example |
---|---|---|---|---|
id | string | Id of filter | true | "table" |
title | string | Title of filter | true | "Table" |
type | string | Type of filter | true | "list" |
datalist | boolean | Is filter has values to select from | false | true |
optional | boolean | Is filter optional | false | false |
secured | boolean | Secured filter values are not available for change by non-owner | false | true |
Read about filter types. Some additional properties should be specified in case when filters should be displayed in special modes, for example JSON or SQL. For example, for highlightText type you also need to specify editorMode for the filter field.
Fields
Fields in Fibery integration app denote either user-supplied fields or fields within an information schema. There are various types of fields that each represent a different type of data. Fields are also used internally to denote information both required and optional for accounts and filters.
Field Types
The table below lists the types of fields referred to in the Fibery integration app REST API.
The first column denotes the string name of the type as expected in the API, the second column the type of underlying
data, the third column any comments and remarks for each field type and the last one show additional options that may
be specified.
type | datatype | comments and remarks | Additional options |
---|---|---|---|
text | string | UTF-8 encoding | - |
number | number | can be decimal, integer, or float | - |
bool | boolean | validates as true/false | - |
password | string | only used in account schemas, rendered as a password input | - |
link | - (used for view) | show link to external source | Need to specify "value" field. It should contain link, for example "http://test.com". |
highlightText | string | text with syntax highlight (json and sql supported for now) | Need to specify "editorMode" field. For now "json" and "sql" are supported. |
datebox | date | single date selector | |
date | date | date selection with support of Date range grammar | - |
oauth | - (used for view) | display OAuth form | - |
list | variant | allows to select value from multiple options | Optional "datalist_requires" can be specified. For example "datalist_requires" = ["owner"] means that fetching the list depends on "owner" field. |
multidropdown | Array |
allows to select multiple values from dropdown list | Optional "datalist_requires" can be specified. For example "datalist_requires" = ["owner"] means that fetching the list depends on "owner" field. |
A Special Note on Dates
Dates, both with and without times, that are parsed as a result of user-input (through the
Date range grammar) are timezone-naive
objects. Dates received from connected
applications may be either aware or naive of timezones and should represent those dates as strings in an
ISO-8601 format.
REST Endpoints
Below are a list of the HTTP endpoints expected in an integration application.
Required
- GET /: returns app information
- POST /validate: performs validation of the account
- POST /api/v1/synchronizer/config: returns synchronizer configuration
- POST /api/v1/synchronizer/schema: returns synchronizer schema
- POST /api/v1/synchronizer/data: returns data
Optional
- GET /logo: returns an image/svg+xml representation of the application's logo
- POST /api/v1/synchronizer/datalist: returns possible options for filter fields
- POST /api/v1/synchronizer/filter/validate: performs filter fields validation
- POST /api/v1/synchronizer/webhooks: attach webhook
- DELETE /api/v1/synchronizer/webhooks: detach webhook
- POST /api/v1/synchronizer/webhooks/verify: verify webhook event
- POST /api/v1/synchronizer/webhooks/transform: transform webhook event into data
GET /
{
"version": "1.0", // string representing the version of your app
"name": "My Super Application", // title of the app
"description": "All your base are belong to us!", // long description
"authentication": [], // list of possible account authentication approaches
"sources": [], // empty error
"responsibleFor": { // app responsibility
"dataSynchronization": true // indicates that app is responsible for data synchronization
}
}
GET "/" endpoint is the main one which returns information about the app. You can find response structure here. Response example:
Authentication Information
{
"authentication": [
{
"id": "basic", // identifier
"name": "Basic Authentication", // user-friendly title
"description": "Just using a username and password", // description
"fields": [ //list of fields to be filled
{
"id": "username", //field identifier
"title": "Username", //friendly name
"description": "Your username, duh!", //description
"type": "text", //field type (text, password, number, etc.)
"optional": true, // is this a optional field?
},
/* ... */
]
}
]
}
The authentication
object includes all account schema information. It informs the Fibery front-end
how to build forms for the end-user to provide required account information. This property is required,
even if your application does not require authentication. At least one authentication object must be provided
within array.
In case when application doesn't require authentication, the fields
array must be omitted.
Read more about fields here.
Important note: if your app provides OAuth capabilities for authentication, the authentication
identifiers must be oauth
and oauth2
for OAuth v1 and OAuth v2, respectively.
Only one authentication type per OAuth version is currently supported.
POST /validate
Incoming body:
{
"id": "basic", // identifier for the account type
"fields": { //list of field values to validate according to schema
"username": "test_user",
"password": "test$user!",
/*...*/
}
}
Success Response:
{"name": "Awesome Account"}
If the account is invalid, the app should return HTTP status 401 (Not Authorized) with a simple JSON object containing an error message:
Failure Response:
{"message": "Your password is incorrect!"}
This endpoint performs account validation when setting up an account for the app and before any actions that uses the account. The incoming payload includes information about the account type to be validated and all fields required:
If the account is valid, the app should return HTTP status 200 with a JSON object containing a friendly name for the account:
Refresh Access Token
Refresh Access Token Request:
{
"id": "oauth2",
"fields": {
"access_token": "xxxx",
"refresh_token": "yyyy",
"expire_on": "2018-01-01"
}
}
Response sample after token refresh:
{
"name": "Awesome account",
"access_token": "new-access-token",
"expire_on": "2020-01-01"
}
In addition this step can be used as a possibility to refresh access token. The incoming payload includes refresh and access token, also it can include expiration datetime. Response should include new access token to override expired one.
POST /api/v1/synchronizer/config
The endpoint returns information about synchronization possibilities based on input parameters. It instructs Fibery about: * Available types * Available filters * Available functionalities
Request
Request example:
{
"account": {
"token": "user-token"
}
}
All input parameters are optional.
Name | Type | Description |
---|---|---|
account | object | selected account's fields |
Response
Response example:
{
"types:": [
{"id": "bug", "name": "Bug"},
{"id": "us", "name": "User Story"},
],
"filters": [
{
"id": "modifiedAfter",
"title": "Modified After",
"optional": true,
"type": "datebox"
}
]
}
Output parameters:
Name | Type | Description |
---|---|---|
types | [{id: string, name: string}] |
supported types list with id and display name |
filters | Array of filters | it is used to help the user to exclude non-required data. |
webhooks | {enabled: true} |
Optional fields that indicates that webhook functionality is supported |
Filter information
The filter
object is used to help the user to exclude non-required data.
Just like other field-like objects, the filter
object is not required. If nothing is provided, users will not be able
to filter out data received from the app.
For more information about filters, please refer to domain and fields types.
POST /api/v1/synchronizer/schema
Request example
{
"types": ["pullrequest", "repository"],
"filter": {
"owner": "fibery",
"repositories": ["fibery/core", "fibery/ui"]
},
"account": {
"token": "token"
}
}
Integration app must provide data schema in advance so Fibery will be able to create approriate types and relations and then be able to maintain them.
It should provide a schema for all requested types. Each type must contain name
and id
field.
In additional there is a reserved field __syncAction
that should be added to the schema if delta synchronization
with possibility of removing items should be supported.
Request
Request contains:
types
- an array of selected type idsfilter
- currently configured filteraccount
- selected account
Response example:
{
"repository": {
"id": {
"type": "id",
"name": "Id"
},
"name": {
"type": "text",
"name": "Name"
},
"url": {
"type": "text",
"name": "Original URL",
"subType": "url"
}
},
"pullrequest": {
"id": {
"type": "id",
"name": "Id"
},
"name": {
"type": "text",
"name": "Name"
},
"repositoryId": {
"type": "text",
"name": "Repository Id",
"relation": {
"cardinality": "many-to-one",
"name": "Repository",
"targetName": "Pull Requests",
"targetType": "repository",
"targetFieldId": "id"
}
},
"__syncAction": {
"type": "text",
"name": "Sync Action"
}
}
}
Response
Includes schema for all requested types
Schema is JSON object where key is field and value if field description. Field description contains:
field | description | type |
---|---|---|
id | Field id | string |
ignore | Is field visible in fields catalog | boolean |
name | Field name | string |
description | Field description | string |
readonly | Disable modify field name and type | boolean |
type | Type of field | "id", "text" ,"number" , "date", "array[text]" |
relation | Relation between types | see relations section |
subType | Optional Fibery sub type | "url" , "integer", "email", "boolean","html", "md", "files", "date-range" |
Relations
Example:
{
"repository": {
"id": {
"type": "id",
"name": "Id"
},
"name": {
"type": "text",
"name": "Name"
},
"url": {
"type": "text",
"name": "Original URL",
"subType": "url"
}
},
"pullrequest": {
"id": {
"type": "id",
"name": "Id"
},
"name": {
"type": "text",
"name": "Name"
},
"repositoryId": {
"type": "text",
"name": "Repository Id",
"relation": {
"cardinality": "many-to-one",
"name": "Repository",
"targetName": "Pull Requests",
"targetType": "repository",
"targetFieldId": "id"
}
}
}
}
relation
field provides a possibility to create a relation between entities in Fibery. It contains following fields:
field | description | type |
---|---|---|
cardinality | Type of relation | "many-to-one", "many-to-many"/ |
name | Name of the field on source side | string |
targetType | Id of target type | string |
targetName | Field name of target side | string |
targetFieldId | Find relation by value from field | string |
Repository will have following fields (example includes only relation fields):
- Pull Requests - Array
Pull Request will have following fields:
- Repository Id - string - this field will be hidden from end user
- Repository - Repository
Reserved field - __syncAction
Sync action is reserved field that won't be visible for end user. The field is used for delta synchronization when entity should be deleted.
POST /api/v1/synchronizer/data
Data endpoint performs actual data retrieving for the specified integration settings. Data retrieving is run by each type independently.
Data synchronization supports:
- pagination
- delta synchronization
Request example:
{
"requestedType": "pullrequest",
"types": ["repository", "pullrequest"],
"filter": {
"owner": "fibery",
"repositories": ["fibery/core", "fibery/ui", "fibery/apps-gallery"]
},
"account": {
"token": "token"
},
"pagination": {
"repositories": ["fibery/ui", "fibery/apps-gallery"]
},
"lastSynchronizedAt": "2020-09-30T09:08:47.074Z"
}
Request
Inbound payload includes following information:
types
- array of selected type idsrequestedType
- currently fetching typeaccount
- account on behalf of data should be fetchedfilter
- currently configured filterslastSynchronizedAt
- OPTIONAL field that indicated when last successful synchronization was runpagination
- OPTIONAL field includes pagination settings that was returned from previous data request
Response example:
{
"items": [
{
"id": "PR_1231",
"name": "Improve performance"
},
{
"id": "PR_1232",
"name": "Fix bugs"
}
],
"pagination": {
"hasNext": true,
"nextPageConfig": {
"repositories": ["fibery/apps-gallery"]
}
},
"synchronizationType": "full"
}
Response
Outboud payload includes:
items
- REQUIRED array of fetched data rowspagination
- OPTIONAL parameter that includes information about paginationhasNext
- boolean attribute that indicates that there are more pages availablenextPageConfig
- object that will be passed inpagination
request body parameter with next page request
synchronizationType
- OPTIONAL parameter with possible valuesdelta
orfull
. It indicates how data will be handled on Fibery side. Ifdelta
is set then only provided changes will be applied. Fibery will be looking for__syncAction
field to identify whether row should be set or removed. If__syncAction
equalREMOVE
then corresponding entity will be removed. Iffull
is set then unsynced data will be removed (if keep unsynced is unchecked).
Errors
If something goes wrong then integration app should respond with corresponding error HTTP status code and error message.
Sample of error about sync failure:
{
"message": "Unable to fetch data."
}
But some errors can be fixed if try to fetch data later. In this case error body should include tryLater
flag with value true
.
Fibery will retry this particular page later on.
Sample of error about limits
{
"message": "Rate limits reached",
"tryLater": true
}
GET /logo
OPTIONAL
The /logo
endpoint is used to provide a SVG representation of a connected application's logo. This endpoint is entirely optional. Valid responses are a HTTP 200 response with a image/svg+xml
content type, a HTTP 204 (No Content) response if there is no logo, or a 302 redirect to another URI containing the logo. If no logo is provided, or an error occurs, the application will be represented with our default app logo.
POST /api/v1/synchronizer/datalist
OPTIONAL
Request body:
{
"types": ["pullrequest", "branch"],
"account": {
"token": "token"
},
"field": "repository",
"dependsOn": {
"owner": "fibery"
}
}
This endpoint performs retrieving datalists from filter fields that marked with datalist
flag.
Request
The inbound payload includes:
types
- an array of selected type idsaccount
- selected accountfield
- name of requested fielddependsOn
- object that contains filter key-value pairs of dependant fields
Response sample:
{
"items": [
{
"title": "fibery/ui",
"value": "124"
},
{
"title": "fibery/core",
"value": "125"
}
]
}
Response
The response from your API should include items
that is a JSON-serialized list of name-value objects:
The title
in each object is what is displayed to the end-user for each value in a combobox
and the value
is what is stored with the filter and what will be passed with subsequent requests
that utilize the user-created filter.
POST /api/v1/synchronizer/filter/validate
OPTIONAL
Request example:
{
"types": ["repository", "pullrequest"],
"filter": {
"owner": "fibery",
"repositories": ["fibery/core", "fibery/ui", "fibery/apps-gallery"]
},
"account": {
"token": "token"
}
}
This endpoint performs filter validation. It can be useful when app doesn't know about what filter value looks like. For example, if your app receives sql query as filter you may want to check is that query is valid.
Request
Request body contains:
types
- array of selected type idsaccount
- account on behalf of data should be fetchedfilter
- currently configured filters
Error response sample:
{"message": "Your filter is incorrect!"}
Response
If the filter is valid, the app should return HTTP status 200 or 204.
If the account is invalid, the app should return HTTP status 400 (Bad request) with a simple JSON object containing an error message:
POST /api/v1/synchronizer/webhooks
OPTIONAL
The endpoint is responsible for installing, updating or reinstalling webhook based on provided parameters.
Request example:
{
"types": ["pullrequest", "repository"],
"filter": {
"owner": "fibery",
"repositories": ["fibery/ui", "fibery/core"]
},
"account": {
"token": "token"
},
"webhook": null
}
Usage cases:
New synchronization source has been created and app supports webhooks. Fibery will attempt to create a webhook.
New settings are applied to synchronization source. Fibery will signal about it. Integration app can do nothing, reply with the same webhook and make updates under the hood or uninstall current webhook and install a new one
Request
Request body contains:
types
- an array of selected type idsfilter
- currently configured filteraccount
- selected accountwebook
- OPTIONAL object that contains currently configured webhook settings
Response example:
{
"id": "webhook_id",
"events": ["create_repo", "delete_repo"]
}
Response
Response is a JSON object with one required field id
. This response object will
be send in next attach webhook event.
DELETE /api/v1/synchronizer/webhooks
OPTIONAL
Request example:
{
"types": ["pullrequest", "repository"],
"filter": {
"owner": "fibery",
"repositories": ["fibery/ui", "fibery/core"]
},
"account": {
"token": "token"
},
"webhook": {
"id": "webhook-id"
}
}
This endpoint is responsible for uninstalling webhook.
Request
Request body contains:
types
- an array of selected type idsfilter
- currently configured filteraccount
- selected accountwebook
- OPTIONAL object that contains currently configured webhook settings
Error response example:
{"message": "Unable to delete webhook"}
Response
If webhook is deleted then it should return HTTP status 200 or 204.
If something goes wrong then app should respond with corresponding error HTTP status code and error message.
POST /api/v1/synchronizer/webhooks/verify
OPTIONAL
{
"params": {
"x-github-id": "1234",
"x-github-signature256": "dsadsa"
},
"payload": {
"action": "create",
"repository": {
"id": "repo1"
}
}
}
This route verifies that webhook event is valid and it can be handled.
Usually it uses SHA verification. The route should respond with id
of webhook.
Request
Request payload includes:
* params
- event parameters that comes from external system.
* payload
- event payload that comes from external system.
Success response:
{
"id": "webhook_id"
}
Response
If verification has passed successfully then response should include webhook
id
field
Failure example:
{
"message": "Invalid signature"
}
Otherwise it should reply with error HTTP status code with error message
POST /api/v1/synchronizer/webhooks/transform
OPTIONAL
Request example:
{
"params": {
"x-github-id": "1234",
"x-github-signature256": "dsadsa"
},
"payload": {
"action": "create",
"repository": {
"id": "repo1"
}
},
"types": ["pullrequest", "repository", "branch"],
"filter": {
"owner": "fibery",
"repositories": []
},
"account": {
"token": "token"
}
}
This route is responsible for converting event payload into data that
can be handled by Fibery. Data is applied by delta
synchronization rules.
Request
Request payload includes:
params
- event parameters that comes from external system.payload
- event payload that comes from external system.types
- an array of selected type idsfilter
- currently configured filteraccount
- selected account
Response
Response:
{
"data": {
"repositories": [
{
"id": "repo1",
"name": "Repo1",
"__syncAction": "SET"
}
],
"branches": [
{
"id": "master",
"name": "master",
"repositoryId": "repo1",
"__syncAction": "SET"
}
]
}
}
Response includes data
object that is a map of data arrays by type id.
OAuth
if your app provides OAuth capabilities for authentication, the authentication identifiers must be oauth
and oauth2
for OAuth v1 and OAuth v2, respectively. Only one authentication type per OAuth version is currently
supported.
OAuth v1
POST /oauth1/v1/authorize
The POST /oauth1/v1/authorize
endpoint performs obtaining request token and secret and generating of authorization url
for OAuth version 1 accounts.
Request body sample:
{
"callback_uri": "https://oauth-svc.fibery.io/callback?state=xxxxxxx"
}
Included with the request is a single body parameter, callback_uri
, which is the redirect URL that the user should be
expected to be redirected to upon successful authentication with the third-party service. callback_uri
includes query
parameter state
that MUST be preserved to be able to complete OAuth flow by Fibery.
Response body sample:
{
"redirect_uri": "https://trello.com/1/OAuthAuthorizeToken?oauth_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&name=TrelloIntegration",
"token": "xxxx",
"secret": "xxxx"
}
Return body should include a redirect_uri
that the user should be forwarded to in order to complete setup, token
and
secret
are granted request token and secret by third-party service. Replies are then POST'ed to
/oauth1/v1/access_token
endpoint.
Note: The OAuth implementation requires the account identifier to be oauth
for OAuth version 1.
Note: If service provider has callback url whitelisting than https://oauth-svc.fibery.io?state=xxxxx
has to be added
to the whitelist.
POST /oauth1/v1/access_token
Request body sample:
{
"fields": {
"access_token": "xxxx",
// token value from authorize step
"access_secret": "xxxxx",
// secret value from authorize step
"callback_uri": "https://oauth-svc.fibery.io?state=xxxxx"
},
"oauth_verifier": "xxxxx"
}
The POST /oauth1/v1/access_token
endpoint performs the final setup and validation of OAuth version 1 accounts.
Information as received from the third party upon redirection to the previously posted callback_uri
are sent to this
endpoint, with other applicable account information, for final setup. The account is then validated and, if successful,
the account is returned; if there is an error, it is to be raised appropriately.
The information that is sent to endpoint includes:
fields.access_token
- request token granted during authorization stepfields.access_secret
- request secret granted during authorization stepfields.callback_uri
- callback uri that is used for user redirectionoauth_verifier
- the verification code received upon accepting on third-party service consent screen.
Response body sample:
{
"access_token": "xxxxxx",
"refresh_token": "xxxxxx",
"expires_on": "2020-01-01T09:53:41.000Z"
}
Response can include any data that will be used to authenticate account and fetch information.
Tip: You can include parameters with refresh_token
and expires_on
and then on validate step
proceed with access token refresh if it is expired or about to expire.
OAuth v2
POST /oauth2/v1/authorize
Request sample
{
"callback_uri": "https://oauth-svc.fibery.io",
"state": "xxxxxx"
}
The POST /oauth2/v1/authorize
endpoint performs the initial setup for OAuth version 2 accounts
using Authorization Code
grant type by generating redirect_uri
based on received parameters.
Request body includes following parameters:
callback_uri
- is the redirect URL that the user should be expected to be redirected to upon successful authentication with the third-party servicestate
- opaque value used by the client to maintain state between request and callback. This value should be included inredirect_uri
to be able to complete OAuth flow by Fibery.
Response example:
{
"redirect_uri": "https://accounts.google.com/o/oauth2/token?state=xxxx&scope=openid+profile+email&client_secret=xxxx&grant_type=authorization_code&redirect_uri=something&code=xxxxx&client_id=xxxxx"
}
Return body should include a redirect_uri
that the user should be forwarded to in order to complete setup.
Replies are then POST'ed to /oauth2/v1/access_token
endpoint.
Note: The OAuth implementation requires the account identifier to be oauth2
for OAuth version 2.
Note: If service provider has callback url whitelisting than https://oauth-svc.fibery.io
has to be added to the
whitelist.
POST /oauth2/v1/access_token
Request body sample:
{
"fields": {
"callback_uri": "https://oauth-svc.fibery.io"
},
"code": "xxxxx"
}
The POST /oauth2/v1/access_token
endpoint performs the final setup and validation of OAuth version 2 accounts.
Information as received from the third party upon redirection to the previously posted callback_uri
are sent to this
endpoint, with other applicable account information, for final setup. The account is then validated and, if successful,
the account is returned; if there is an error, it is to be raised appropriately.
The information that is sent to endpoint includes:
fields.callback_uri
- callback uri that is used for user redirectioncode
- the authorization code received from the authorization server during redirect oncallback_uri
Response body sample:
{
"access_token": "xxxxxx",
"refresh_token": "xxxxxx",
"expires_on": "2020-01-01T09:53:41.000Z"
}
Response can include any data that will be used to authenticate account and fetch information.
Tip: You can include parameters with refresh_token
and expires_on
and then on validate step
proceed with access token refresh if it is expired or about to expire.
Date range grammar
In order to make filtering by dates and date ranges easier on the end-user, Fibery integration app uses a custom domain specific language to represent dates and date ranges. The syntax allows a user to specify either a static date or a dynamic date range using plain English statements.
Arbitrary Dates
Dates can be easily and quickly specified without any knowledge of the DSL by simply inserting a date in either DD-MMMM-YYYY, DD MMMM YYYY, DD-MM-YYYY or DD MM YYYY format.
The Date Spec DSL also includes various keyword elements that can be used to quickly insert dynamic values. In your date specification you may specify today and yesterday to get the appropriate date value. These date values will always parse in relation to the current day, allowing you to specify a date or date range of, or relative to, these dates.
Periods
Periods in the date spec DSL is where the majority of the magic happens. The DSL understands various time periods, as outlined in the chart below. These time periods can be referenced either aligned to their appropriate boundaries (calendar months, full weeks, quarters, etc.) or arbitrarily aligned to a particular day (the past month, past week, etc.).
Period | Aligned value | Arbitrary value |
---|---|---|
day | a day | a day |
week | iso week (Mon - Sun) | 7 days |
month | calendar month (1st day start) | 30 days |
quarter | calendar quarter (Q1, Q2, etc.) | 90 days |
year | calendar year (Jan 1 - Dec 31) | 365 days |
Grammar usage
Periods can be referenced, as mentioned earlier, either aligned to an appropriate date boundary or arbitrarily aligned. When using an aligned period, the period fits to the calendar representation of that period. For example, an aligned month starts on the 1st day and ends on the last, whereas an arbitrary month would represent 30 days starting or ending on a particular day.
LAST and THIS statements
The last keyword creates a period aligned to the last period represented. Consider the following examples:
DSL statement | value |
---|---|
last week | the last full calendar week starting from Monday |
last month` | the last full month, starting from the 1st |
last quarter | the last full quarter |
last year | the last full year |
User can specify period interval exactly. For example, last 3 weeks
, last 8 months
and etc.
this keyword is an alias for last and can be used instead of it.
PREVIOUS statement
The previous keyword creates a range whose starts with first day of the period before the current period and continues for period length. Consider the following examples:
DSL statement | value |
---|---|
previous week | starts at 12:00:00 AM on the first day of the week before the current week and continues for seven days |
previous month | starts at 12:00:00 AM on the first day of the month before the current month and continues for all the days of that month |
previous quarter | starts at 12:00:00 AM on the first day of the calendar quarter before the current calendar quarter and continues to the end of that quarter |
previous year | starts at 12:00:00 AM on January 1 of the year before the current year and continues through the end of December 31 of that year |
User can specify period interval exactly. For example, previous 3 weeks
, previous 8 months
and etc.
BETWEEN statements
The between keyword allows you to specify a date range or minimum and maximum values to filter on with more power and precision. The full format for this statement is BETWEEN date AND date
where both dates are some representation of a date (this can be an arbitrary date, keyword or period).
ago keyword can be used inside between to specify date more precisely.
Consider the following examples:
DSL statement | value |
---|---|
between 6 quarters ago and 2 months ago | starts at 12:00:00 AM on the first day of the calendar quarter 6 quarters before the current calendar quarter and continues until to the first day of the calendar month 2 months before the current calendar month |
between 1 January 2016 and today | between Jan 1, 2013 and today |
FROM and TO statements
from and to allow user to specify ranges without minimum or maximum values.
Consider the following examples:
DSL statement | value |
---|---|
from 1 Jan 2010 | starts at 1 Jan 2010 till now |
to 1 Jan 2012 | ends with 1 Jan 2012 |
Receiving Parsed Dates
The date range grammar parses an input date with each use of a filter that utilizes a date-type field. These dates are provided to application connectors as a dictionary object representing a date range. User-input dates and date ranges are timezone-naive.
Date always is parsed as date range. Even if user enters a single date in a field, this date will be parsed and further represented as a dictionary object with _min
equal to start of the date and _max
equal to end of the date.
If a user enters a date range with no defined minimum or maximum, such as would be the case when using the FROM
or TO
keywords, _min
and _max
keys will not be present and the dictionary will have only _min
or _max
as its single key-value pair.
_min
and _max
represented as a string in ISO-8601 format.
How to test and debug
Expose local instance
Expose local instance to world:
brew install ngrok
ngrok http 8080
It is possible to run your app on local machine and make the app's url publicly available by using tools like ngrok. Then you will have an ability to debug the app locally after adding it Fibery apps gallery. Don't forget to remove the app from Fibery integration apps catalog after testing.
Integration tests
const request = require(`supertest`);
const app = require(`./app`);
const assert = require(`assert`);
const _ = require(`lodash`);
describe(`integration app suite`, function () {
it(`should have the logo`, async () => {
await request(app).get(`/logo`)
.expect(200)
.expect(`Content-Type`, /svg/);
});
it(`should have app config`, async () => {
const {body: appConfig} = await request(app).get(`/`)
.expect(200).expect(`Content-Type`, /json/);
assert.equal(appConfig.name, `Public Holidays`);
assert.match(appConfig.description, /public holidays/);
assert.equal(appConfig.responsibleFor.dataSynchronization, true);
});
it(`should have validate end-point`, async () => {
const {body: {name}} = await request(app).post(`/validate`)
.expect(200).expect(`Content-Type`, /json/);
assert.equal(name, `Public`);
});
it(`should have synchronization config`, async () => {
const {body: {types, filters}} = await request(app)
.post(`/api/v1/synchronizer/config`)
.expect(200)
.expect(`Content-Type`, /json/);
assert.equal(types.length, 1);
assert.equal(filters.length, 3);
});
it(`should have schema holidays type defined`, async () => {
const {body: {holiday}} = await request(app)
.post(`/api/v1/synchronizer/schema`)
.send()
.expect(200)
.expect(`Content-Type`, /json/);
assert.deepEqual(holiday.id, {name: `Id`, type: `id`});
});
it(`should return data for CY`, async () => {
const {body: {items}} = await request(app)
.post(`/api/v1/synchronizer/data`)
.send({
requestedType: `holiday`,
filter: {
countries: [`CY`],
}
}).expect(200).expect(`Content-Type`, /json/);
assert.equal(items.length > 0, true);
const holiday = items[0];
assert.equal(holiday.id.length > 0, true);
assert.equal(holiday.name.length > 0, true);
});
it(`should return data for BY and 2020 year only`, async () => {
const {body: {items}} = await request(app)
.post(`/api/v1/synchronizer/data`)
.send({
requestedType: `holiday`,
filter: {
countries: [`BY`],
from: 2020,
to: 2020
}
}).expect(200).expect(`Content-Type`, /json/);
assert.equal(items.length > 0, true);
const holidaysOtherThan2020 = _.filter(items, (i) =>
new Date(i.date).getFullYear() !== 2020);
assert.equal(holidaysOtherThan2020.length > 0, false);
});
});
It is recommended to create integration tests before adding your custom app to Fibery apps gallery. Check some tests I created for holidays app.
Tutorial: Simple app
In the tutorial we will implement the simplest app with token authentication. We will use NodeJS and Express framework, but you can use any other programming language. Find source code repository here.
Requirements
const users = {
token1: {
name: `Dad Sholler`,
data: {
flower: [
{
id: `47fd45cf-5a07-40aa-9ee4-a4258832154a`,
name: `Rose`,
},
{
id: `4f3afc75-2fb9-4ff8-b26f-ab4bf2f470a3`,
name: `Lily`,
},
{
id: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
name: `Tulip`,
},
],
regionPrice: [
{
id: `56c50696-1d9f-4e4d-9678-448017d25474`,
flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
price: 10,
name: `Eastern Europe`,
},
{
id: `503e5efb-7650-4c3a-85e9-f4ebc23adfd5`,
name: `Western Europe`,
flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
price: 15,
},
{
id: `87ccd5ef-ed3f-49db-9bcc-c9d1daf91744`,
name: `Eastern Europe`,
price: 20,
flowerId: `47fd45cf-5a07-40aa-9ee4-a4258832154a`,
},
],
},
},
token2: {
name: `Ben Dreamer`,
data: {
flower: [
{
id: `4f3afc75-2fb9-4ff8-b26f-ab4bf2f470a3`,
name: `Lily`,
},
{
id: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
name: `Tulip`,
},
],
regionPrice: [
{
id: `c3352e8c-e62c-4e26-9a8b-852c1a3d2435`,
flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
price: 10,
name: `East Coast`,
},
{
id: `7a581588-d5fc-46aa-a084-8e2b28f3d6e5`,
name: `West Coast`,
flowerId: `d7525fc1-1979-4cfb-ba32-0dfab9280b24`,
price: 15,
},
{
id: `6e510195-41e3-499b-9a76-9ad928720882`,
name: `Asia`,
price: 20,
flowerId: `4f3afc75-2fb9-4ff8-b26f-ab4bf2f470a3`,
},
],
},
},
};
To create simple app you need to implement following endpoints:
- getting app information: GET /
- validate account: POST /validate
- getting synchronizer configuration: POST /api/v1/synchronizer/config
- getting schema: POST /api/v1/synchronizer/schema
- fetching data POST /api/v1/synchronizer/data
Let's implement them one by one. But first let's define dataset
Getting app information
app.get(`/`, (req, res) => {
res.json({
id: 'integration-sample-app',
name: 'Integration Sample App',
version: `1.0.0`,
type: 'crunch',
description: 'Integration sample app.',
authentication: [
{
description: 'Provide Token',
name: 'Token Authentication',
id: 'token',
fields: [
{
type: 'text',
description: 'Personal Token',
id: 'token',
},
],
},
],
sources: [],
responsibleFor: {
dataSynchronization: true,
},
});
});
App information should have the structure. Let's implement it. We can see that the app require token authentication and responsible for data synchronization. You can find more information about the endpoint here.
Validate account
app.post(`/validate`, (req, res) => {
const user = users[req.body.fields.token];
if (user) {
return res.json({
name: user.name,
});
}
res.status(401).json({message: `Unauthorized`});
});
Since authentication is required we should run an authentication and return corresponding user name. You can find more information about the endpoint here.
Getting sync configuration
app.post(`/api/v1/synchronizer/config`, (req, res) => {
res.json({
types: [
{id: `flower`, name: `Flower`},
{id: `regionPrice`, name: `Region Price`},
],
filters: [],
});
});
This is basic a scenario and we don't need any dynamic. So we will return a static configuration with no filters. You can find more information about the endpoint here.
Getting schema
const schema = {
flower: {
id: {
name: `Id`,
type: `id`,
},
name: {
name: `Name`,
type: `text`,
},
},
regionPrice: {
id: {
name: `Id`,
type: `id`,
},
name: {
name: `Name`,
type: `text`,
},
price: {
name: `Price`,
type: `number`,
},
flowerId: {
name: `Flower Id`,
type: `text`,
relation: {
cardinality: `many-to-one`,
name: `Flower`,
targetName: `Region Prices`,
targetType: `flower`,
targetFieldId: `id`,
},
},
},
};
app.post(`/api/v1/synchronizer/schema`, (req, res) => {
res.json(
req.body.types.reduce((acc, type) => {
acc[type] = schema[type];
return acc;
}, {}),
);
});
We should provide schema for selected types. You can find more information about the endpoint here.
Fetching data
app.post(`/api/v1/synchronizer/data`, (req, res) => {
const {requestedType, account} = req.body;
return res.json({
items: users[account.token].data[requestedType],
});
});
The endpoint receives requested type, accounts, filter and set of selected types and should return actual data. You can find more information about the endpoint here. And that's it! Our app is ready to use. See full example here.
Tutorial: Holidays App
In this article we will show how to create simple integration app which does not require any authentication. We intend to create a public holidays app which will sync data about holidays for selected countries. The holidays service Nager.Date will be used to retrieve holidays.
The source code can be found here
App Information
const appConfig = require(`./config.app.json`);
app.get(`/`, (req, res) => res.json(appConfig));
Response (config.app.json):
{
"id": "holidays-app",
"name": "Public Holidays",
"version": "1.0.1",
"description": "Integrate data about public holidays into Fibery",
"authentication": [
{
"id": "public",
"name": "Public Access",
"description": "There is no any authentication required",
"fields": [
]
}
],
"sources": [
],
"responsibleFor": {
"dataSynchronization": true
}
}
Every integration should have the configuration which describes what the app is doing and the authentication methods.
The app configuration should be accessible at GET /
endpoint and should be publicly available.
NOTE: All mentioned properties are required.
Since I don't want my app be authenticated I didn't provide any fields for "Public Access" node in authentication. It means that any user will be able to connect their account to the public holidays app. Find an example with token authentication here .
Validate Account
app.post(`/validate`, (req, res) => res.json({name: `Public`}));
This endpoint is responsible for app account validation. It is required to be implemented. Let's just send back the name of account without any authentication since we are creating an app with public access.
Sync configuration
const syncConfig = require(`./config.sync.json`);
app.post(`/api/v1/synchronizer/config`, (req, res) => res.json(syncConfig));
Response(config.sync.json):
{
"types": [
{
"id": "holiday",
"name": "Public Holiday"
}
],
"filters": [
{
"id": "countries",
"title": "Countries",
"datalist": true,
"optional": false,
"type": "multidropdown"
},
{
"id": "from",
"type": "number",
"title": "Start Year (by default previous year used)",
"optional": true
},
{
"id": "to",
"type": "number",
"title": "End Year (by default current year used)",
"optional": true
}
]
}
The way data is synchronised should be described.
The endpoint is POST /api/v1/synchronizer/config
types - responsible for describing types which will be synced. For the holidays app it is just one type with id " holidays" and name "Public Holidays". It means that only one integration Fibery database will be created in the space, with the name "Public Holidays".
filters - contains information on how the type can be filtered. In our case there is a multi dropdown ('countries') which is required and marked as datalist. It means that options for this dropdown should be retrieved from app and special end-point should be implemented for that. We have two numeric filters from and to which are optional and can be used to filter holidays by years.
Find information about filters here.
Countries datalist
app.post(`/api/v1/synchronizer/datalist`, wrap(async (req, res) => {
const countries = await (got(`https://date.nager.at/api/v3/AvailableCountries`).json());
const items = countries.map((row) => ({title: row.name, value: row.countryCode}));
res.json({items});
}));
Endpoint POST /api/v1/synchronizer/datalist
should be implemented if synchronizer filters has dropdown marked
as "datalist": true
. Since we have countries multi dropdown filter which should contain countries it is required to
implement the mentioned endpoint as well.
For example part of countries response will look like this:
{
"items": [
...
{
"title": "Poland",
"value": "PL"
},
{
"title": "Belarus",
"value": "BY"
},
{
"title": "Cyprus",
"value": "CY"
},
{
"title": "Denmark",
"value": "DK"
},
{
"title": "Russia",
"value": "RU"
}
]
}
NOTE: For this app, only the list of countries is returned since our config has only one data list. In the case
where there are several data lists then we will need to retrieve "field" from request body which will contain an id of
the requested list. The response should be formed as an array of items
where every element contains title
and value
properties.
Schema
const schema = require(`./schema.json`);
app.post(`/api/v1/synchronizer/schema`, (req, res) => res.json(schema));
schema.json
{
"holiday": {
"id": {
"name": "Id",
"type": "id"
},
"name": {
"name": "Name",
"type": "text"
},
"date": {
"name": "Date",
"type": "date"
},
"countryCode": {
"name": "Country Code",
"type": "text"
}
}
}
POST /api/v1/synchronizer/schema
endpoint should return the data schema of the app. In our case it should contain only
one root element "holiday" named after the id of holiday type in sync configuration above. Find the documentation about
schemas here.
NOTE: Every schema node should have id
and name
elements defined.
Data
const getYearRange = filter => {
let fromYear = parseInt(filter.from);
let toYear = parseInt(filter.to);
if (_.isNaN(fromYear)) {
fromYear = new Date().getFullYear() - 1;
}
if (_.isNaN(toYear)) {
toYear = new Date().getFullYear();
}
const yearRange = [];
while (fromYear <= toYear) {
yearRange.push(fromYear);
fromYear++;
}
return yearRange;
};
app.post(`/api/v1/synchronizer/data`, wrap(async (req, res) => {
const {requestedType, filter} = req.body;
if (requestedType !== `holiday`) {
throw new Error(`Only holidays database can be synchronized`);
}
if (_.isEmpty(filter.countries)) {
throw new Error(`Countries filter should be specified`);
}
const {countries} = filter;
const yearRange = getYearRange(filter);
const items = [];
for (const country of countries) {
for (const year of yearRange) {
const url = `https://date.nager.at/api/v3/PublicHolidays/${year}/${country}`;
console.log(url);
(await (got(url).json())).forEach((item) => {
item.id = uuid(JSON.stringify(item));
items.push(item);
});
}
}
return res.json({items});
}));
The data endpoint POST /api/v1/synchronizer/data
is responsible for retrieving data. There is no paging needed in case
of our app, so the data is returned according to selected countries and years interval. The requestedType
and filter
can be retrieved from the request body. The response should be returned as array in items
element.
Tutorial: Notion App
This section is created in order to provide help on creating complex integration app with dynamic data schema, non-primitive data synchronization (for examples files) and oauth2 authentication. The source code (node.js) can be found in official Fibery repository which contains the implementation of integrating Notion databases into Fibery databases. Demo databases can be found here
App Configuration
Route in app.js
app.get(`/`, (req, res) => res.json(connector()));
Returns the description of the app and possible ways to be authenticated in Notion.
connector.config.js:
const config = require(`./config`);
const ApiKeyAuthentication = {
description: `Please provide notion authentication`,
name: `Token`,
id: `key`,
fields: [
{
type: `password`,
name: `Integration Token`,
description: `Provide Notion API Integration Token`,
id: `key`,
},
{
type: `link`,
value: `https://www.notion.so/help/create-integrations-with-the-notion-api`,
description: `We need to have your Notion Integration Token to synchronize the data.`,
id: `key-link`,
name: `Read how to create integration, grant access and create token here...`,
},
],
};
const OAuth2 = {
id: 'oauth2',
name: 'OAuth v2 Authentication',
description: 'OAuth v2-based authentication and authorization for access to Notion',
fields: [
{
title: 'callback_uri',
description: 'OAuth post-auth redirect URI',
type: 'oauth',
id: 'callback_uri',
},
],
};
const getAuthenticationStrategies = () => {
return [OAuth2, ApiKeyAuthentication];
};
module.exports.connector = () => ({
id: `notion-app`,
name: `Notion`,
version: config.version,
website: `https://notion.com`,
description: `More than a doc. Or a table. Customize Notion to work the way you do.`,
authentication: getAuthenticationStrategies(),
responsibleFor: {
dataSynchronization: true,
},
sources: [],
});
As you see there are two authentication ways are defined:
OAuth2
Hardcoded "oauth2"
should be used as id
in case you would like to implement OAuth2 support in integration app.
Token Authentication
You may use special field type: "link"
in order to provide url for external resource where the user can get more
info. Use type:"password"
for tokens or other text fields which need to be secured.
Token Authorization
Route (app.js):
app.post(`/validate`, (req, res) => promiseToResponse(res, notion.validate(_.get(req, `body.fields`) || req.body)));
Request Body:
{
"id": "key",
"fields": {
"app": "620a3c9baec5dd25794fed7a",
"auth": "key",
"owner": "620a3c46cf7154924cf442cb",
"key": "MY TOKEN",
"enabled": true
}
}
Notion call (the name of account is returned):
module.exports.validate = async (account) => {
const client = getNotionClient(account);
const me = await client.users.me();
return {name: me.name}; //reponse should include the name of user account
};
The implementation of token authentication is the simplest way to implement. We always used it for testing and
development since it is not required UI interaction. The request contains id
of auth and user provided values. In our
case it is key
. Other fields are appended by system and can be ignored.
OAuth 2
app.js
app.post('/oauth2/v1/authorize', (req, res) => {
try {
const {callback_uri: callbackUri, state} = req.body;
const redirectUri = oauth.getAuthorizeUrl(callbackUri, state);
res.json({redirect_uri: redirectUri});
} catch (err) {
res.status(401).json({message: `Unauthorized`});
}
});
app.post('/oauth2/v1/access_token', async (req, res) => {
try {
const tokens = await oauth.getAccessToken(req.body.code, req.body.fields.callback_uri);
res.json(tokens);
} catch (err) {
res.status(401).json({message: 'Unauthorized'});
}
});
OAuth 2 is a bit more complex and requires several routes to be implemented. The POST /oauth2/v1/authorize
endpoint
performs the initial setup for OAuth version 2 accounts using Authorization Code
grant type by
generating redirect_uri
based on received parameters. Read more here.
The POST /oauth2/v1/access_token
endpoint performs the final setup and validation of OAuth version 2 accounts.
Information as received from the third party upon redirection to the previously posted callback_uri
are sent to this
endpoint, with other applicable account information, for final setup. Read more here.
oauth.js
const got = require(`got`);
const CLIENT_ID = process.env.ENV_CLIENT_ID;
const CLIENT_SECRET = process.env.ENV_CLIENT_SECRET;
module.exports = {
getAuthorizeUrl: (callbackUri, state) => {
const queryParams = {
state,
redirect_uri: callbackUri,
response_type: 'code',
client_id: CLIENT_ID,
owner: `user`,
};
const queryParamsStr = Object.keys(queryParams)
.map((key) => `${encodeURIComponent(key)}=${encodeURIComponent(queryParams[key])}`)
.join(`&`);
return `https://api.notion.com/v1/oauth/authorize?${queryParamsStr}`;
},
getAccessToken: async (code, callbackUri) => {
const tokens = await got.post(`https://api.notion.com/v1/oauth/token`, {
resolveBodyOnly: true,
headers: {
"Authorization": `Basic ${Buffer.from(`${CLIENT_ID}:${CLIENT_SECRET}`).toString('base64')}`,
},
json: {
code,
redirect_uri: callbackUri,
grant_type: `authorization_code`,
},
}).json();
return {access_token: tokens.access_token};
},
};
The implementation of oauth is pretty similar for many services and Notion is not exclusion here. Find the code of
oauth.js in the right code panel. access_token
will be passed into /validate
for validating token in future calls.
Synchronizer configuration
app.js (route)
app.post(`/api/v1/synchronizer/config`, (req, res) => {
if (_.isEmpty(req.body.account)) {
throw new Error(`account should be provided`);
}
promiseToResponse(res, notion.config(req.body));
});
notion.api.js
const getDatabases = async ({account, pageSize = 1000}) => {
const client = getNotionClient(account);
let hasNext = true;
let start_cursor = null;
const databases = [];
while (hasNext) {
const args = {
page_size: pageSize, filter: {
value: `database`, property: `object`,
}
};
if (start_cursor) {
args.start_cursor = start_cursor;
}
const {results, has_more, next_cursor} = await client.search(args);
results.forEach((db) => databases.push(db));
hasNext = has_more;
start_cursor = next_cursor;
}
return databases;
};
const getDatabaseItem = (db) => {
const name = _.get(db, `title[0].plain_text`, `Noname`).replace(/[^\w ]+/g, ``).trim();
return {id: db.id, name};
};
module.exports.config = async ({account, pageSize}) => {
const databases = await getDatabases({account, pageSize});
const dbItems = databases.map((db) => getDatabaseItem(db)).concat({id: `user`, name: `User`});
return {types: dbItems, filters: []};
};
Response example
{
"types": [
{
"id": "f4642444-220c-439d-85d6-378ddff3d510",
"name": "Features"
},
{
"id": "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
"name": "Tasks"
},
{
"id": "user",
"name": "User"
}
],
"filters": []
}
This endpoint returns types which should be synced to Fibery databases. In Notion case it is the list of databases.
Static user
type is added. Check how the configuration response looks like
for Notion Demo.
Schema of synchronization
app.js (schema route)
app.post(`/api/v1/synchronizer/schema`, (req, res) => promiseToResponse(res, notion.schema(req.body)));
notion.api.js
module.exports.schema = async ({account, types}) => {
const databases = await getDatabases({account});
const mapDatabasesById = _.keyBy(databases, `id`);
const schema = {};
types.forEach((id) => {
if (id === `user`) {
schema.user = userSchema;
return;
}
const db = mapDatabasesById[id];
if (_.isEmpty(db)) {
throw new Error(`Database with id "${id}" is not found`);
}
schema[id] = createSchemaFromDatabase(db);
});
cleanRelationsDuplication(schema);
return schema;
};
Request example:
{
"account": {
"_id": "620a4396aec5dd672c4fed83",
"access_token": "USER-TOKEN",
"app": "620a3c9baec5dd25794fed7a",
"auth": "oauth2",
"owner": "620a3c46cf7154924cf442cb",
"enabled": true,
"name": "Fibery Developer",
"masterAccountId": null,
"lastUpdatedOn": "2022-02-21T09:45:37.802Z"
},
"filter": {},
"types": [
"f4642444-220c-439d-85d6-378ddff3d510",
"3bd058e6-a71c-4e9a-8480-a76810ae38d3",
"user"
]
}
Response example:
{
"f4642444-220c-439d-85d6-378ddff3d510": {
"id": {
"type": "id",
"name": "Id"
},
"archived": {
"type": "text",
"name": "Archived",
"subType": "boolean"
},
"created_time": {
"type": "date",
"name": "Created On"
},
"last_edited_time": {
"type": "date",
"name": "Last Edited On"
},
"__notion_link": {
"type": "text",
"name": "Notion Link",
"subType": "url"
},
"related to tasks (column)": {
"name": "Related to Tasks (Column) Ref",
"type": "text",
"relation": {
"cardinality": "many-to-many",
"targetFieldId": "id",
"name": "Related to Tasks (Column)",
"targetName": "Feature",
"targetType": "3bd058e6-a71c-4e9a-8480-a76810ae38d3"
}
},
"tags": {
"name": "Tags",
"type": "array[text]"
},
"due date": {
"name": "Due Date",
"type": "date"
},
"name": {
"name": "Name",
"type": "text"
}
},
"3bd058e6-a71c-4e9a-8480-a76810ae38d3": {
"id": {
"type": "id",
"name": "Id"
},
"archived": {
"type": "text",
"name": "Archived",
"subType": "boolean"
},
"created_time": {
"type": "date",
"name": "Created On"
},
"last_edited_time": {
"type": "date",
"name": "Last Edited On"
},
"__notion_link": {
"type": "text",
"name": "Notion Link",
"subType": "url"
},
"status": {
"name": "Status",
"type": "text"
},
"assignees": {
"name": "Assignees Ref",
"type": "array[text]",
"relation": {
"cardinality": "many-to-many",
"targetType": "user",
"targetFieldId": "id",
"name": "Assignees",
"targetName": "Tasks (Assignees Ref)"
}
},
"specs": {
"name": "Specs",
"type": "array[text]",
"subType": "file"
},
"link to site": {
"name": "Link to site",
"type": "text",
"subType": "url"
},
"name": {
"name": "Name",
"type": "text"
}
},
"user": {
"id": {
"type": "id",
"name": "Id",
"path": "id"
},
"name": {
"type": "text",
"name": "Name",
"path": "name"
},
"type": {
"type": "text",
"name": "Type",
"path": "type"
},
"email": {
"type": "text",
"name": "Email",
"subType": "email"
}
}
}
The schema which describes fields and relations should be provided for each sync type. Find full implementation here. It is not easy thing to implement since we are talking about dynamic data in Notion databases.
It can be noticed that almost any field from Notion database can be mapped into Fibery field using subType
attribute.
Relations can be mapped as well. Rich text can be sent as html
or md
by defining corresponding type="text"
and subType="md" or "html"
.
Note: Relation between databases(types) should be declared only once. Double declarations for relations will lead to
duplication of relations in Fibery databases. We implemented the function cleanRelationsDuplication
in order to remove
redundant relation declarations from schema fields.
Files field mapping:
"specs": {
"name": "Specs",
"type": "array[text]",
"subType": "file"
}
Data route
app.js
app.post(`/api/v1/synchronizer/data`, (req, res) => promiseToResponse(res, notion.data(req.body)));
notion.api.js (paging support)
const getValue = (row, {path, arrayPath, subPath = ``}) => {
let v = null;
const paths = _.isArray(path) ? path : [path];
paths.forEach((p) => {
if (!_.isUndefined(v) && !_.isNull(v)) {
return;
}
v = _.get(row, p);
});
if (!_.isEmpty(subPath) && _.isObject(v)) {
return getValue(v, {path: subPath});
}
if (!_.isEmpty(arrayPath) && _.isArray(v)) {
return v.map((element) => getValue(element, {path: arrayPath}));
}
if (_.isObject(v)) {
if (v.start) {
return v.start;
}
if (v.end) {
return v.end;
}
if (v.type) {
return v[v.type];
}
return JSON.stringify(v);
}
return v;
};
const processItem = ({schema, item}) => {
const r = {};
_.keys(schema).forEach((id) => {
const schemaValue = schema[id];
r[id] = getValue(item, schemaValue);
});
return r;
};
const resolveSchema = async ({pagination, client, requestedType}) => {
if (pagination && pagination.schema) {
return pagination.schema;
}
if (requestedType === `user`) {
return userSchema;
}
return createSchemaFromDatabase(await client.databases.retrieve({database_id: requestedType}));
};
const createArgs = ({pageSize, pagination, requestedType}) => {
const args = {
page_size: pageSize,
};
if (!_.isEmpty(pagination) && !_.isEmpty(pagination.start_cursor)) {
args.start_cursor = pagination.start_cursor;
}
if (requestedType !== `user`) {
args.database_id = requestedType;
}
return args;
};
module.exports.data = async ({account, requestedType, pageSize = 1000, pagination}) => {
const client = getNotionClient(account);
const schema = await resolveSchema({pagination, client, requestedType});
const args = createArgs({pageSize, pagination, requestedType});
const data = requestedType !== `user`
? await client.databases.query(args)
: await client.users.list(args);
const {results, next_cursor, has_more} = data;
return {
items: results.map((item) => processItem({account, schema, item})),
"pagination": {
"hasNext": has_more,
"nextPageConfig": {
start_cursor: next_cursor,
schema: has_more ? schema : null,
},
},
};
};
Request example:
{
"filter": {},
"types": [
"f4642444-220c-439d-85d6-378ddff3d510",
"3bd058e6-a71c-4e9a-8480-a76810ae38d3",
"user"
],
"requestedType": "3bd058e6-a71c-4e9a-8480-a76810ae38d3",
"account": {
"_id": "620a4396aec5dd672c4fed83",
"access_token": "USER-TOKEN",
"app": "620a3c9baec5dd25794fed7a",
"auth": "oauth2",
"owner": "620a3c46cf7154924cf442cb",
"enabled": true,
"name": "Fibery Developer",
"masterAccountId": null,
"lastUpdatedOn": "2022-02-21T13:30:51.350Z"
},
"lastSynchronizedAt": null,
"pagination": null
}
Response example:
{
"items": [
{
"id": "4455580b-000b-4313-8128-f1ca2d2dec34",
"archived": false,
"created_time": "2022-02-14T11:28:00.000Z",
"last_edited_time": "2022-02-14T11:30:00.000Z",
"__notion_link": "https://www.notion.so/Login-Page-4455580b000b43138128f1ca2d2dec34",
"related to tasks (column)": [
"b829daf3-bae5-40a0-a090-56a30f240a28"
],
"tags": [
"Urgent"
],
"due date": "2022-02-24",
"name": [
"Login Page"
]
},
{
"id": "9b3dff11-582b-498a-ba9b-571827ab3ca7",
"archived": false,
"created_time": "2022-02-14T11:28:00.000Z",
"last_edited_time": "2022-02-14T11:29:00.000Z",
"__notion_link": "https://www.notion.so/Home-Page-9b3dff11582b498aba9b571827ab3ca7",
"related to tasks (column)": [
"987a714b-0b7e-4b03-bdaf-c0efc5d522fb",
"539a4d0e-6871-434b-a5cb-619f5bd5a911"
],
"tags": [
"Important",
"Urgent"
],
"due date": "2022-02-14",
"name": [
"Home Page"
]
}
],
"pagination": {
"hasNext": false,
"nextPageConfig": {
"start_cursor": null,
"schema": null
}
}
}
Notion supports paged output, so it is handy to fetch data page by page. The response should include pagination
node
with hasNext
equals to true
or false
and nextPageConfig
(next page configuration) which will be included with
the future request as pagination
.
You may notice that we have included schema
into nextPageConfig
(pagination config). It is not required and it is
done as an optimization in order to save some between pages fetching on schema resolving. In other words the pagination
can be used as a context cache between page calls.
Source Code
The source code of Notion integration can be found in our public repository as well as other examples. Notion app is used in production and can be tried by following integrate link in your database editor.