Platform Replicate (previously Platform Sync) lets you set up a data source connector for any external database on any external system for which we do not have a live connector.

While it is a bit more technical to configure, it allows you to get around certain platform limitations, such as our 50,000-row limit on system data sources, since Platform Replicate data sources can have any number of rows.

This article discusses when to use it, its pros and cons, and how to set it up. Please note that we assume you understand data sources, database systems, and API endpoints, as we use these terms.

Overview

Some key benefits of using Platform Replicate data sources in your projects.

When the rows become large, the app still has performance limits — these vary across Android and iOS and depend heavily on device specifications.

As a rule of thumb, performance is more affected by the overall file size of the data source than by the number of rows.

For example, 5,000 rows containing 100kb of text per row can be expected to run slower than 50,000 rows with a total of 1kb per row.

Additionally, using “heavy” data sources across many screens that can be navigated sequentially by the user will degrade compounding performance, as those large data sources can progressively accumulate in device memory.

With Platform Replicate, you must test the app to find the best row count and screen mix for your specific data set and usage requirements.

Due to the technical complexity and requirements of implementing a Platform Replicate data source, you are assumed to have software development resources to create the web service that the Platform Replicate connector needs.

How it all Works

When the app discovers that a Data Source has a Platform Replicate connector, it performs an HTTP GET request against the target web service URL.

The GET request contains the External ID of the requested Data Source and other identifying information.

You MUST expose your web service endpoint as SSL-secured HTTPS to protect app requests.

The request includes default query-string parameters: your account’s Private Token (helpful for authenticating the request) and the user’s email address (useful for filtering rows based on the user’s access).

The app will issue GET requests sequentially (one request per Data Source) to your service whenever it attempts a data synchronization.

While the exact way you configure your Platform Replicate endpoint is entirely up to you—i.e., you can use PHP, Python, Node.js, or any number of programming languages to achieve your desired result—a C# example that outlines the request and response structures as simple classes can be found towards the end of the article.

Even if you are not using C#, reviewing the properties and structure in code form can be very helpful in getting started with your own Platform Replicate service.

Setting up a Platform Replicate Data Source Connector

For test purposes, we recommend you use a REST API development service like – https://www.mockable.io

1. Create a new data source or select an existing one.
2. On the Rows page of your Data Source, manually define and organize your Data Source’s columns to match the output of your web service.
   
   For any column references to work, the order of the columns must match what your web service returns.
   
   If you change the column order in your web service response at any stage, you must update the columns on the Rows page to match.
3. On the Settings page of your Data Source, ensure the External ID field is set to a value that identifies this Data Source.
   
   Remember this External ID value. You will need it later in this document when you work through the GET web service details.
4. Then, add a Platform Replicate connector on the Settings page via the “Add Connector” button.
   
   An External ID is required, and the address of your web service endpoint in the Target GET URL.
5. You can also set a global Data Source Replicate URL for your organization (defined in Organization Setup > Global Data Source URL) and specify that the connector uses it. This allows the reuse of the same service endpoint across multiple data sources.

GET Request Essentials

The Organization Setup screen contains most settings for integrating with your Platform Replicate API, including the Integration Key (Private Token) and Provider ID (Company ID).

Your web service MUST return a response in the following structure. Failure to comply with this format will prevent your app from downloading rows.
Your response MUST be in JSON format.

We also support gzip and deflate compression on responses. This is recommended for large data sets, since text data compresses very well, conserving mobile data and massively reducing download time.

Our compression support is identified through the “Accept-Encoding” header set by the app on its REST requests.

Your web service/server should encode the entire response stream using gzip or deflate, which our app code will recognize.

This is usually an automatic behavior at the web server level, though you should check that your server configuration has enabled JSON compression.

This is beyond our support scope regarding how to apply response encoding – you’ll need to confirm this with your software developers.

If you use Microsoft IIS as your web server, this article may help you check your compression settings.

This service may also help identify if your server response is indeed being compressed.

GET Request Syntax

The app issues GET requests to your service, passing the values below as query string parameters.

This is the format of a typical request made by the app to your Platform Replicate endpoint, with user@somewhere.com as the logged-in user.

GET https://yourdomain.com/xx/rowsearch?email=user@somewhere.com&integrationkey=xxxxxxxxxxxx&providerid=1&id=JOBS&lastupdated=2024-02-12T14:24:09

The parameters passed on the request are detailed below:

Given that this connector is REST-based, you can test your web service directly via your web browser using a REST plugin like the Postman plugin for Google Chrome.

In this example, we assume you have set the “Target GET URL” on your Platform Replicate Connector to https://yourdomain.com/xx/rowsearch.

We also assume that the Data Source has an External ID of “JOBS”.

GET Response Syntax

Your endpoint MUST return the data in the JSON format specified below:

{
  "DataSources": [
    {
      "Id": "JOBS",
      "Rows": [
        [
          "10011",
          "10011 - Test Generic Project",
          "TEST",
          "12345AA",
          "19 Nov 2013",
          "12 Somewhere Road"
        ],
        [
          "ZDS654",
          "ZDS654 - Maintenance Costs to Excavator",
          "TEST",
          "",
          "27 Nov 2013",
          "24 Somewhere Else Road"
        ],
        [
          "ZDF662",
          "ZDF662 - Maintenance Costs for Hitachi Excavator",
          "TEST2",
          "",
          "28 Nov 2013",
          "182 Another Place, Somewhere"
        ]
      ],
      "NewRows": [],
      "DeletedRows": []
    }
  ],
  "LastUpdated": "2024-02-11T15:45:50.4818098+10:00"
}

GET Response Details

Below is a detailed description of how each node on the JSON response body returned by your API endpoint should be formatted.

The “Rows” array in the GET response from your endpoint (detailed in the table below) is a collection of values (Val), each representing the individual column value for that Row.

Every Row must contain at least two values (Val). The first Val should be the unique identifier or key used to identify this specific row of values returned by your endpoint.

The first value (Val) is also critical as the key for performing incremental deletes through the DeletedRows field mentioned above.

The second value (Val) should be the row’s default displayable label text. You can override this default by configuring the Display Options field in the Data Source “Settings” page.

Each value in the “Rows” array corresponds to a column in the dataset your endpoint returns. Below is a summary of how each returned value should be formatted.

Your web service should return a 400-level HTTP status if the GET fails.

Example Code

C# example that outlines the request and response structures as simple classes.

Even if you are not using C#, reviewing the properties and structure in code form can be very helpful in getting started with your own Platform Replicate service.

using System;
using System.Collections.Generic;

namespace Example.HostedGET
{
    // Data Transfer Object (DTO) classes for web services linked to Hosted GET connectors on Data Sources.
    // A Hosted GET service exposes the rows for a Data Source from an external system/database.
    // The service must accept the request payload described in the DRows class below, 
    // and must respond with the payload described in the DRowsResponse class.

    /// <summary>
    /// Request DTO class for calls made by the app to Hosted GET web services.
    /// </summary>
    public class DRowsRequest
    {
        /// <summary>
        /// Comma separated list of External Ids for each Data Source that the app wants to synchronise.  
        /// You set the External Id on the Data Source via the App Builder -> Data Sources -> Settings page in the platform.
        /// </summary>
        public string Ids { get; set; }

        /// <summary>
        /// Comma separated list of last updated values for each external id in Ids that the app wants to synchronise.  
        /// Values are listed in the same order as Ids and will contain a blank value if the related data source have not been requested before.
        /// Use this to implement incremental app updates, passing back just the new/updated and delete rows since the time the app checked in. 
        /// This is recommended for large Data Sources to save server resources and prevent exhausting mobile data allocations.
        /// </summary>
        public string LastUpdateds { get; set; }

        /// <summary>
        /// Your unique Company Id, found on the Organisation Setup page in the platform
        /// </summary>
        public int ProviderId { get; set; }

        /// <summary>
        /// Your unique Integration Key, found on the Organisation Setup page in the platform
        /// </summary>
        public string IntegrationKey { get; set; }

        /// <summary>
        /// Email address of the logged in app user making the request
        /// </summary>
        public string Email { get; set; }
    }

    /// <summary>
    /// Response DTO class for calls made by the app to Hosted GET web services.
    /// </summary>
    public class DRowsResponse
    {
        /// <summary>
        /// Collection containing the DataSources that have been requested.
        /// </summary>
        public DSources DataSources { get; set; }

        /// <summary>
        /// The last time that the rows for this Data Source were updated.  
        /// We recommend that this be the current date and time on your host server at the time of processing the request.
        /// This value is stored by the app and sent in the LastUpdateds field of the next Row Search request made by the app.
        /// This gives a central point of time for incremental updates in particular, since the app will echo your server times rather than use its own unreliable device time.
        /// </summary>
        public DateTime LastUpdated { get; set; }
    }

    public class DSource
    {
        /// <summary>
        /// The External Id of the DataSource for which the associated Row data applies.  Must match the External Id configured in the Data Source -> Settings page  on the platform.
        /// </summary>
        public string Id { get; set; }

        /// <summary>
        /// A collection of Rows that will completely replace all current Data Source rows on app.
        /// If you specify Rows, then any values in NewRows and DeletedRows will be ignored.
        /// </summary>
        public Rows Rows { get; set; }

        /// <summary>
        /// A collection of Rows that should be added/updated onto the existing Data Source rows on the app.
        /// Use this field to perform incremental inserts/updates of your Data Source.
        /// </summary>
        public Rows NewRows { get; set; }

        /// <summary>
        /// A collection of Row Items that should be deleted from the existing Data Source rows on the app.
        /// Use this field to perform incremental deletes of your Data Source.
        /// Each Row should contain a single Val element that is your unique identifier for the Row to delete.
        /// </summary>
        public Rows DeletedRows { get; set; }
    }
	
    public class DSources : List<DSource> { }
	
    public class Rows : List<Row> { }

    public class Row : List<string> { }

}

Troubleshooting / Testing

Once you have set up a Platform Replicate connector on your Data Source, the Rows page for that Data Source will automatically attempt to pull back rows from the given GET URL.

The Rows page makes the same request as the app, including your current logged-in user.

This is a quick and easy way to see if your Platform Replicate web service understands the GET request and returns rows in the correct format.

What do I do when Platform Replicate works on my IOS app but not Android?

This is almost certainly due to how Android treats SSL certificates – essentially, Android is more strict about SSL certificate chains than iOS.

SSL certificate chains involve the primary SSL certificate for your web service endpoint domain and any intermediate certificates issued by your Certificate Authority.

If you attempt to connect to a web address with an incomplete SSL chain, Android will fail the connection.

If your Android devices cannot sync, you can test if the issue is related to an incomplete SSL chain by browsing your Platform Replicate service URL from your default Android web browser.

You will likely get a connection error with text such as “the certificate authority is invalid”.

You can further verify your SSL configuration using a testing service like SSL Labs:

https://www.ssllabs.com/ssltest

If your SSL installation has issues, you will see warnings such as “chain incomplete” on the SSL Labs test results.

To resolve the issue, ensure the Platform Replicate web service server has the intermediate certificates from your Certificate Authority installed.

For example, if you use GoDaddy, the best place to start would be here:

https://support.godaddy.com/help/article/868/what-is-an-intermediate-certificate?countrysite=au

Once you have the correct certificate chain, the Platform Replicate data sources will sync to your Android devices since they can now connect to your web service.

You can confirm the certificate chain by checking the Platform Replicate URL in your Android web browser. When resolved, it should no longer show certificate errors.