Integrating with the JourneyApps Platform

mike_barnes · April 21, 2021, 5:34pm

A how-to guide that provides information about the options available when integrating external systems with the JourneyApps platform.

There are many ways to integrate external systems/client applications with applications developed on the JourneyApps application development platform. This article will cover each of the options available to developers and will provide examples (written in JavaScript and XML).

The main integration options available to developers include:

Backend API
CloudCode Webhook
CloudCode Webtasks
Oplog
OpSQL
On-Device
External Webhooks

Backend API

Each backend instance that is created on Oxide automatically generates a REST API that exposes the models defined in the app schema.xml as well as the data.

For more detailed information on the Backend API, please see this section of the documentation.

Rate Limits

Before we look at the use case it is important to note that rate limits apply when requesting data from the Backend API, here is a detailed description of these limits:

REST API - sustained maximum requests per second	20
REST API - burst capacity	1200

Use Case

This method of integration with JourneyApps is best suited for use cases where third-party applications/client applications need to perform CRUD operations against the backend instance. A good example of this would be syncing master data with the JourneyApps platform that needs to synchronize on devices.

Authentication

The BackendAPI supports two methods of authenticating HTTP requests. Please see examples on authentication and authorization in this article.

Examples

To create a concrete example, let’s use the following data model as an example:

<model name="user" label="User">
    <field name="name" label="Name" type="text:name"/>
    <field name="id_number" label="Id Number" type="text" />
    <has-many model="warehouse" name="warehouses" />
    <display>{name}</display>
</model>

<model name="warehouse" label="Warehouse">
    <field name="name" label="Name" type="text" />
    <field name="registration_number" label="Registration Number" type="number" />
    <field name="closed" label="Closed" type="boolean" />
    <field name="barcode" label="Barcode" type="attachment" media="any" />
    <belongs-to model="user" />
    <has-many model="section" name="sections" />
    <display>{name}</display>
</model>

<model name="section" label="Section">
    <field name="name" label="Name" type="text" />
    <belongs-to model="warehouse" />
    <has-many model="stock_item" name="stock_items" />
    <display>{name}</display>
</model>

<model name="stock_item" label="Stock Item">
    <field name="name" label="Name" type="text" />
    <belongs-to model="section" />
    <display>{name}</display>
</model>

List Objects

Fetch a list of all objects for the specified model. The API limits the number of records returned (1000 objects per request) and you’ll need to build paging logic using limiting and skipping query params.

Here is a simple example of how to get a list of warehouses:

const fetch = require("node-fetch");

async function warehouses() {
    const BASE_URL = "https://run-testing-us.journeyapps.com/api/v4/60144fda076011449fbe15c0";
    const TOKEN = "YOUR API TOKEN";
    const options = {
        headers: {
            method: 'GET',
            Authorization: `Bearer ${TOKEN}`
        }
    };

    const warehouseUrl = `${BASE_URL}/objects/warehouse.json`;
    const response = await fetch(warehouseUrl, options);
    if (response.status !== 200) {
        throw new Error(`Request failed: ${response.status}`)
    }
    const jsonResponse = await response.json();
    const warehouses = jsonResponse.objects;
    console.log(warehouses);
    return warehouses; // Returns an array of warehouse objects
}

For more information on this, please see our documentation.

Query Objects

Fetch a list of objects from the Backend API based on filter conditions/fields that are defined in your data model.

Here is an example of fetching warehouse objects based on a name query:

async function warehousesByName (name) {
   const BASE_URL = "https://run-testing-us.journeyapps.com/api/v4/60144fda076011449fbe15c0"
   const TOKEN = "YOUR API TOKEN";

   const options = {
       headers: {
           method: 'GET',
           Authorization: `Bearer ${TOKEN}`
       }
   };

   const warehouseUrl = `${BASE_URL}/objects/warehouse.json?query[name]=${name}`;

   const response = await fetch(warehouseUrl, options);
   if(response.status !== 200) {
       throw new Error(`Request failed: ${response.status}`)
   }

   const warehouses = await response.json();
   console.log(warehouses);
   return warehouses;
}

Notice that we have included the query[name] query parameter to specify which field we want to query on. We can also add operators into the query parameter, e.g., contains. We will then update the query parameter to query[name.contains]=”Ad” to search for all warehouses where the name field contains “Ad”.

For other operators such as not equal, please visit this article on querying objects.

Create Object

Thus far we’ve covered using an HTTP GET to list and query objects, let’s look at how to create a new instance of a warehouse object using the Backend API via HTTP POST.

Here is a snippet of code that creates a new warehouse object and sets the corresponding belongs-to relationship via the API:

async function createWarehouse () {
   const BASE_URL = "https://run-testing-us.journeyapps.com/api/v4/60144fda076011449fbe15c0"
   const TOKEN = "YOUR API TOKEN";

   // The model is nested in the body as warehouse
   const body = {
       warehouse: {
           name: "A new warehouse created via the API",
           registration_number: 28221,
           closed: false,
           user_id: "user_guid"
       }
   }

   const options = {
       headers: {
           "method": 'POST',
           "Authorization": `Bearer ${TOKEN}`,
           "Content-Type": "application/json"
       },
       body: JSON.stringify(body)
   };

   const warehouseUrl = `${BASE_URL}/objects/warehouse.json`;

   const response = await fetch(warehouseUrl, options);
   if(response.status !== 200) {
       throw new Error(`Request failed: ${response.status}`)
   }

   const warehouse = await response.json();
   console.log(warehouse);
   return warehouse;
}

It’s important to understand how data model fields are represented in the Backend API. For example, when creating an object and setting a boolean field you can set the field to true/false, but when you fetch objects the boolean field is displayed differently, e.g. {"key": true, "display": "Yes"}. Please make sure you familiarize yourself with the fields’ representations.

Update Object

When updating an object, you’ll need to specify the ID of the object you are updating in the url. You can get the ID from a query HTTP request, as exemplified in the Query Objects or List Objects section above.

Here is a sample function which updates the name and closed boolean of a single object instance using an HTTP PATCH. If you want to update the entire object you’d use a HTTP PUT. Please note you will mostly be using HTTP PATCH for making updates to existing objects. HTTP PUT is used when you want to replace an existing object entirely.

async function updateWarehouse(id, name, closed) {
    const BASE_URL = "https://run-testing-us.journeyapps.com/api/v4/60144fda076011449fbe15c0"
    const TOKEN = "YOUR API TOKEN";

    const body = {
        warehouse: {
            closed: closed,
            name: name
        }
    };

    const options = {
        headers: {
            "method": 'PATCH',
            "Authorization": `Bearer ${TOKEN}`,
            "Content-Type": "application/json"
        },
        body: JSON.stringify(body)
    };

    const warehouseUrl = `${BASE_URL}/objects/warehouse/${id}.json`;

    const response = await fetch(warehouseUrl, options);
    if (response.status !== 200) {
        throw new Error(`Request failed: ${response.status}`)
    }

    const warehouse = await response.json();
    console.log(warehouse);
    return warehouse;
}

Delete Object

To delete an object you can use the HTTP DELETE method to delete a specific instance of an object.

async function deleteWarehouse(id) {
    const BASE_URL = "https://run-testing-us.journeyapps.com/api/v4/60144fda076011449fbe15c0"
    const TOKEN = "YOUR API TOKEN";

    const options = {
        headers: {
            "method": 'DELETE',
            "Authorization": `Bearer ${TOKEN}`,
        }
    };

    const warehouseUrl = `${BASE_URL}/objects/warehouse/${id}.json`;

    const response = await fetch(warehouseUrl, options);
    if (response.status !== 200) {
        throw new Error(`Request failed: ${response.status}`)
    }

    const warehouse = await response.json();
    console.log(warehouse);
    return warehouse;
}

Batch Operations

The JourneyApps Backend API also allows you to specify multiple operations for objects in a batch.

Use Case

Using batch operations is very useful for use cases where you need to update/create/delete large sets of data. This is commonly used for master data management or bulk uploads of data from other development environments.

Examples

When using batch an array of operations must be created. Within these operations you’ll need to specify a method. These methods inform the server what needs to be done with the objects, here is the list of available methods:

POST: Create a new object
PUT: Replace an object instance
PATCH: Partial update of an object instance
DELETE: Delete an object

async function createWarehouses(warehouses) {
    const BASE_URL = "https://run-testing-us.journeyapps.com/api/v4/60144fda076011449fbe15c0";
    const TOKEN = "YOUR API TOKEN";

    const options = {
        headers: {
            "method": 'POST',
            "Authorization": `Bearer ${TOKEN}`,
            "Content-Type": "application/json"
        },
        body: null
    };

    let batchRequest = {
        operations: []
    };
    for (let i = 0; i < warehouses.length; i++) {
        batchRequest.operations.push({
            method: "post",
            object: {
                type: "warehouse",
                name: warehouses[i].name
            }
        });
    }

    options.body = JSON.stringify(batchRequest);

    const url = `${BASE_URL}/batch.json`;

    const response = await fetch(url, options);
    if (response.status !== 200) {
        throw new Error(`Request failed: ${response.status}`)
    }
}

What is important to know with the example above is that if you do not format the objects correctly the server will fail without performing any of the operations. Optionally you can add a key named stop_on_error and set the value of that to true. What this means is that all operations prior to an error operation will execute and take effect on the app database.

Let’s update the batchRequest in the example code above to exemplify this:

let batchRequest = {
    stop_on_error: true,
    operations: []
};
for (let i = 0; i < warehouses.length; i++) {
    batchRequest.operations.push({
        method: "post",
        object: {
            type: "warehouse",
            name: warehouses[i].name
        }
    });
}

CloudCode Webhook

CloudCode webhooks allow you to reach out to HTTP endpoints when objects are created or updated. The webhooks are defined on your data model objects and invoke a CloudCode task when objects are created or updated.

Use Case

CloudCode Webhooks are ideal for situations where payloads need to be posted to APIs listening for events that occur in the app. A good example of this is when a new order is created and the order details need to be posted to an endpoint.

Examples

In this example we’ll configure our model, set up a CloudCode task and use a HTTP post to send the content of the model which is created to an endpoint. First, we’ll need to define the webhook with specific attributes that indicate the webhooks needs to invoke a CloudCode task. Here is an example data model definition:

<model name="lead" label="Lead">
    <field name="first_name" label="First Name" type="text" />
    <field name="last_name" label="last_name" type="text" />
    <field name="email" label="Email" type="text:email" />
    <field name="processed" label="Processed" type="boolean" />
    <field name="file" label="File" type="photo" />

    <webhook type="ready" name="external_send_email" action="process_lead" receiver="cloudcode" />
    
    <display>{first_name} {last_name}</display>
</model>

Notice that we added the action attribute which is used to specify which CloudCode task to invoke and the action attribute which instructs the Webhook to use CloudCode. The type of the webhook has been set to ready. This means that the CloudCode task is invoked when a new instance of the lead object is created. An alternative would be to set the type to update. This causes the webhook to be invoked every time an instance of a lead object is updated.

Next, let’s create a new CloudCode task which will post the newly created lead.

By default CloudCode includes the node-fetch, we’ll use this package to perform the HTTP request.

export async function run(event) {
    let lead;
    if (!event) {
        // Event will be null if you use the Test Task button
        // Fetch an existing object to test or create one
        lead = await DB.lead.first("INSERT ID HERE");
    } else {
        // The event contains the lead object
        // Along with other diagnostics info
        lead = event.object;
    }

    // Create a new object which is a copy of the lead
    // object and strp away the JourneyApps platforrm properties
    const body = Object.assign({}, lead);

    const url = "https://yourdomain/resource";

    const config = {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            "Authorization": "Bearer <Token>"
        },
        body: JSON.stringify(body)
    }

    try {
        const response = await fetch(url, config);
        if (response.status !== 200) {
            console.log("Request failed", response.status);
        } else {
            lead.processed = true;
            await lead.save();
        }
    } catch (err) {
        console.log("Failed to execute fetch request", err);
    }
}

CloudCode Webtasks

CloudCode Webtasks allow you to create custom HTTP endpoints that client applications can consume. This differs from the Backend API in that it’s not automatically generated off of the data model definition, instead a function for each HTTP method is exposed and it is up to you to populate and implement them as needed.

Use Case

This is great for a use case where an external system requires specialized endpoints to either receive or send data to and the Backend API cannot be used. This method gives the developer a lot more control over how the APIs work, as opposed to the Backend API which is pre-built. It allows developers to use custom authentication mechanisms, other than Basic Authentication or tokens It also allows for custom validation, custom error handling, and custom logic. This is generally the recommended route for incoming integrations.

Authentication

The CloudCode Webtasks do not use the basic or token authentication that is provided by the backend instance (as used by the Backend API). Authentication is handled by the developer and an authenticate method is created by default when a new Webtask is created in Oxide. This function must return either access.authorized(); or or access.unauthorized(). This function is always invoked before any other function in the Webtask.

Examples

For this example, let’s say our external system sends a list of warehouses and the sections that belong to a warehouse.

HTTP POST

Create a new CloudCode task and select the Webtask template

By default you’ll have the following functions in your CloudCode task:

Authenticate
A function that is executed prior to any other function, here you need to implement any authentication you require for the Webtask
Get
Exposes an HTTP GET function which you need to implement. For this example, we’ll update this to return a message that this method is not implemented.

export async function get({ params, request, response, authContext }) {
    console.log("New GET received, returning 501. Not Implemented");
    response.status(501);
}

Run

This is not used by the Webtask, however, you can use this to perform quick and easy tests from Oxide.

Let’s create a new function to accept HTTP POST requests. This function will receive an array of warehouses and each warehouse will contain an array of sections. The function will validate the params (passed by the client application) and then create the objects. Once completed it sets the response status of 200 (OK).

// HTTP POST request
export async function post({ params, request, response, authContext }) {
    const warehouses = params.warehouses;
    if (!warehouses) {
        // Return HTTP 400 - Bad Request
        response.status(400);
        return;
    }
    // Create a batch to process large amounts of data
    const batch = new DB.Batch();
    for (let warehouse of warehouses) {
        const warehouseObj = DB.warehouse.create();
        warehouseObj.name = warehouse.name ? warehouse.name : null;
        batch.save(warehouseObj);
        for (let section of warehouse.sections) {
            const sectionObj = DB.section.create();
            sectionObj.name = section.name;
            sectionObj.warehouse_id = warehouseObj.id;
            batch.save(sectionObj);
        }
    }
    await batch.execute();
    response.status(200);
}

Oplog

The JourneyApps OpLog API (operations log) provides an ordered log of all create, update and delete operations that occurred on the cloud data store for one of your JourneyApps applications.

Use Case

The Oplog is great for cases where external systems poll the Oplog and then sync the changes to another data source. The Oplog is useful for use cases where an audit trail of changes in data is required.

Authentication

The Oplog API supports two methods of authenticating HTTP requests. Please see examples on authentication and authorization in this article.

Examples

In this example, let’s retrieve the top 10 changes from the Oplog.

HTTP GET

To get the most recent changes we need to include a query param, namely tail=true. Below is an example function that fetches the records.

async function getChanges() {
    const BASE_URL = "https://run-testing-us.journeyapps.com/api/v4/60144fda076011449fbe15c0"
    const TOKEN = "YOUR API TOKEN";

    const options = {
        headers: {
            "method": 'GET',
            "Authorization": `Bearer ${TOKEN}`,
        }
    };

    const url = `${BASE_URL}/oplog.json?limit=10&tail=true`;

    const response = await fetch(url, options);
    if (response.status !== 200) {
        throw new Error(`Request failed: ${response.status}`)
    }

    const changes = await response.json();
    console.log(changes);
    return changes;
}

OpSQL

OpSQL or the SQL Data Replication Pipeline provides an out-of-the-box managed solution that replicates any JourneyApps Cloud DB to an MS SQL database hosted on JourneyApps’ Azure Cloud. The replication occurs automatically, in real-time, and no manual intervention is required.

Use Case

OpSQL is ideal for use cases where business intelligence dashboards or reports are required, and it is easier to build those on top of a SQL database as opposed to building them directly off of the JourneyApps Cloud DB. Other ASP.NET applications can also integrate directly with the SQL database, and developers can also easily implement SQL to SQL replication of the managed OpSQL instance into their own SQL DB.

On-Device

The JourneyApps runtime allows developers to integrate with HTTP endpoints directly on the device using the fetch API, without the need of server-side interactions, as is the case with the Backend API, CloudCode Webtasks, the Oplog or OpSQL. On-device integration will not allow you to expose API endpoints, however, it can reach out to APIs.

Use Case

On-device integration is ideally used in use cases where a set of APIs are only accessible within an internal network or intranet and app users need to retrieve data from the network.

For example, a user needs to retrieve master data from REST APIs on an intranet.

Examples

Let’s use an example where we first retrieve the US population data and then POST data to a demo endpoint.

HTTP GET

Let use the Data USA API to get the population count by year and nation, then save this to DB objects in JourneyApps. Each population object will contain a nation, year and population count. We’ll use a button component to initiate the request and bind the getData function below to the on-press attribute of the button i.e. when the user selects the button the request will be made.

function getData() {
    var url = "https://datausa.io/api/data?drilldowns=Nation&measures=Population";
    var options = {
        method: "GET",
        headers: {
            "Content-Type": "application/json"
        }
    };

    return fetch(url, options)
        .then(function (response) {
            if (response.status < 200 || response.status >= 300) {
            // Request failed
            throw new Error("Failed to make network request: [" + response.status + "]");
            } else {
                return response.json();
            }
        }).then(function (json) {
            var populationArrary = json.data;
            console.log(JSON.stringify(populationArrary, null, 2));
            for (var i = 0; i < populationArrary.length; i++) {
                notification.info("Syncing " + (i + 1) + "/" + populationArrary.length);
                var population = DB.population.first("nation = ? and year = ?", populationArrary[i].Nation, populationArrary[i].Year);
                if (!population) {
                    population = DB.population.create();
                    population.year = populationArrary[i].Year;
                    population.nation = populationArrary[i].Nation;
                    population.count = populationArrary[i].Population;
                    population.save();
                }
            }
            return notification.info("Request completed");
        }).caught(function (error) {
            dialog("Error fetching population data", error.message);
        });
}

HTTP POST

Now that we have a GET working, let’s POST an array of population objects stored in the JourneyApps DB to a sample API. We’ll use a request bin to host the endpoint. Similar to the get we’ll use a button UI component to execute the POST request.

function payload() {
    var populationList = DB.population.all().toArray();
    return {
        count: populationList.length,
        sent_at: new Date(),
        data: populationList
    };
}

function postData() {
    var body = payload();
    var url = "https://envx80sr2b3rh.x.pipedream.net/";
    var options = {
        method: "POST",
        headers: {
            "Content-Type": "application/json"
        },
        body: JSON.stringify(body)
    };
    return fetch(url, options)
        .then(function (response) {
                    if (response.status < 200 || response.status >= 300) {
                // Request failed
                throw new Error("Failed to make network request: [" + response.status + "]");
            } else {
                return response.json();
            }
        }).then(function (json) {
            return notification.info("Request completed");
        }).caught(function (error) {
            dialog("Error sending population data", error.message);
        });
}

Once the user selects the button the data is read and it populates objects from the DB, builds a request body, and then finally the data is posted. We can then inspect the result of this request and confirm that it works as expected.

External Webhooks

The final integration option we’ll cover in this article that is available on the JourneyApps platform is external Webhooks. JourneyApps supports webhooks that allow remote endpoints to be invoked when data is changed in the database. These webhooks are configured on a model and can be triggered when conditions are met.

Use Case

Let’s say we have a form that needs to be completed by a user, i.e., a user captures his/her first name, last name, email address, and proof of identity. Once completed and the user has uploaded the proof of identity, a Webhook can be triggered that sends an email confirming that the user has completed the process.

Examples

As mentioned, above Webhooks can be triggered when a condition on a JourneyApps object is met, so let’s start by configuring the Webhook and conditions in our data model/schema.xml. The condition, in this case, is that the file attachment must be uploaded before the Webhook can trigger.

<model name="lead" label="Lead">
    <field name="first_name" label="First Name" type="text" />
    <field name="last_name" label="last_name" type="text" />
    <field name="email" label="Email" type="text:email" />
    <field name="file" label="File" type="photo" />

    <webhook type="ready" name="external_send_email">
        <field name="file" required="true" embed="true" state="uploaded" />
    </webhook>
    
    <display>{first_name} {last_name}</display>
</model>

Notice that we added the webhook node and that the type is set to ready. This means the webhook will only execute, once, when a new instance is created. The alternative type is updated, this means the webhook will execute every time the object is edited and saved. We’ve also instructed the webhook to only execute when the file field is uploaded and present on the server.

Next, we need an endpoint that will listen for requests from the JourneyApps Webhook.

You can visit this sample endpoint the request bin that will run when we create objects.

Once you have your endpoint ready, you’ll need to visit the backend instance and configure the URL that the data will be posted. To do so visit the Manage Webhooks page in the Data Browser. Click the top right arrow to open up the menu below and select Manage Webhooks. The Manage Webhooks page allows you to view the details and health of your Webhooks.

Next, you’ll need to configure the URL by selecting the pencil icon on the webhook (as defined in your data model) in the User Hooks list, paste in the URL, and save e.g.

Click on the Enable checkbox button to make sure that the Webhook is active.

Finally, we can test this process by manually creating an object in the backend data browser and check the request bin to see if the data was sent.

As you can see, the Webhook was triggered and the object property in the response contains the object values from JourneyApps.

forumfred · April 21, 2021, 5:54pm

Very thorough, thanks

Topic		Replies	Views
How to connect API to oxide in journeyApps General Q&A	6	888	June 30, 2021
Backend API and Oplog API HTTP Authentication Guides	1	467	April 21, 2021
Notices: Backend API rate limits for all apps; and Mapbox API deprecation Announcements and Notices	1	482	January 26, 2021
Database General Q&A	2	296	September 8, 2021
How do I send a raw webhook to an external system / URL General Q&A webhooks , integration	1	356	February 17, 2021

Integrating with the JourneyApps Platform

Backend API

Rate Limits

Use Case

Authentication

Examples

List Objects

Query Objects

Create Object

Update Object

Delete Object

Batch Operations

Use Case

Examples

CloudCode Webhook

Use Case

Examples

CloudCode Webtasks

Use Case

Authentication

Examples

HTTP POST

Oplog

Use Case

Authentication

Examples

HTTP GET

OpSQL

Use Case

On-Device

Use Case

Examples

HTTP GET

HTTP POST

External Webhooks

Use Case

Examples

Related Topics