Difference between revisions of "Data Import Service"

Revision as of 16:09, 22 December 2020

The Data Import Service is a interface consisting of a web service, hosted on an IIS service, which takes data pushed to Ubik and imports it efficiently. The heart of the service is a shared module called the Import System, which can be used to import data from whatever source they come, easing future interface developments. The system is highly customizable via Ubik Plugins to fit most use cases.

In detail, the service consists of three major parts:

• Web Service
  • Customizable Authentication System
• Customizable Import System

Web Service

The Web Service contains the externally callable endpoint, containing the type reference of the import object in the URL and the data in the body:

This request also expects some Authentication data, which are completely dependent on the customized implementation provided via plugins.

If you want to develop an interface to get data from your 3rd party system to Ubik, you might want to consider using this system as well. It's easy to set up and requires minimal configuration on the Ubik side.

Authentication System

There is currently no default implementation, so in order to get the Web Service running, one plugin containing a custom ImportValidator has to be provided. It contains a single method to override, Validate, which contains the headers of the incoming requests to validate the Authorization header. There is also a possibility to set the outgoing headers for the response, for example, to return a token that makes authentication easier next time.

Import System

The Import System is the heart of the entire system. It is responsible for creating Ubik objects out of raw data, via proxies.

In its ideal use case, the interface runs the following steps:

• Creating ImportObject derivate out of the raw data
• Loading ImportProcessor derivate supporting the type of ImportObject
• Validating the ImportObject with the ImportProcessor
• Writing the ImportObject to a new or existing Proxy instance
• Saving the Proxy

At this time, the data are successfully saved in Ubik and the interface returns a status and object information. However, there's some more stuff going on in the background:

• Async bulk-import of Proxy to create Ubik object
• Once finished and new items are queued, a new bulk-import will launch. If no import is queued, the customizable post processing method will be called.

Similarly to the Authentication System, this system can be customized via Plugins as well. Providing a custom ImportProcessor derivate with the target type set to a custom ImportObject derivate makes this processor available to be used to fit a use-case.

The Import System can be used by itself too, it's easy to integrate it to different interfaces. It can be made running synchronously as well, to be potentially even more efficient for specific custom implementations.