Operator APIs | Data Ingestion

Data Ingestion API Authentication

Data Ingestion is a product that allows NetRefer clients to use REST principles to ingest data into the PMP.

It is possible to ingest data in real-time sending a single event or in batch.

To access the APIs, clients should have an appropriate account registered with NetRefer.

This account provides access to specific NetRefer products, which in our case is API.

Also, clients must provide a Subscription Key allowing access to Data Ingestion APIs.

The authentication flow is depicted conceptually in the diagram below.

Picture

Using Active Directory to request a JWT

OAuth 2.0 is used to get access to the Data Ingestion API.

To authenticate within NetRefer and to call the Data Ingestion API, developers must request JWT, providing the following parameters:

  • Grant type: password

  • Client ID: will be provided to the client while onboarding to Data Ingestion. Example of id is: 5a324196-7727-45fd-8ac3-1436cd541d76

  • Access Token URL: https://login.microsoftonline.com/0e0b50ac-5253-4347-b528-251b2f17cfc6/oauth2/v2.0/token

  • Client Secret: will be provided to the client while onboarding to Data Ingestion

  • Username: will be provided to the client while onboarding to Data Ingestion

  • Password: will be provided to the client while onboarding to Data Ingestion

  • Scope: api://5a324196-7727-45fd-8ac3-1436cd541d76/Events.Injestion

Once all the requirements are met, execute an HTTP POST call to "Access Token URL" to get the JWT.

In the API request, you have to add an "Authorization" header with the value:

  • Bearer JWT-value-from-NetRefer-Data Ingestion

An example of a request using different languages can be found on the APIs page of this portal.

Authorization to use Data Ingestion product

To authenticate access to the Data Ingestion product, HTTP requests must include an additional header:

  • Ocp-Apim-Subscription-Key: will be provided to the client during onboarding to Data Ingestion. Example: baea25a9f0fe4687845575c07e539a9e9

A full example of the Data Ingestion API call is illustrated in the diagram below.

Data Ingestion API usage

All Data Ingestion APIs are exposed as REST endpoints.

Request and response models

Request data models are presented and described via the OpenAPI specification.

These specifications can be downloaded from the portal and the developers can build their own code proxies.

In the Developer Portal, you can also find an example of request\response objects and their schema:

Response Status Codes

Web API uses the following response status codes, as defined in the RFC 2616 and RFC 6585:

Data Ingestion Data Model and Error Codes

Learn more about common Data Ingestion Developer error codes and how to resolve them.

Error codes describe events’ validation statuses of the DATA INGESTION API.

If you don't find an error code listed here, please refer to the API-specific error codes in their respective documentation pages.

All events are sent in batches but processed individually. If all events are valid, then 200 will be returned. If some of the events are not valid, then 207 will be returned. If all the events are not valid, then 400 will be returned.

 

Additional validation for all event types:

  • multiple events with the same EventID must not be processed

  • JSON with unproper multiple structures must not be processed

What to do if Data Ingestion is having issues? 

Data Ingestion is designed on the most cutting-edge technology stack and platform.

The API is constantly available, allowing for real-time data input.

If the API is unavailable for any reason, or if the status codes received are one of the following, Data Ingestion customers should follow these guidelines:

  1. Contact your consultant and/or NetRefer Support if you have 401, 403, 404, or 500 (if it appears for a long period of time).

  2. We recommend using the retry strategy to trigger your endpoints until they succeed in providing real-time data intake (and to be proactive with 500 error code processing). There may be a one-, three-, five-, or ten-minute time gap between calls, or any other interval the customer chooses.