Comparison of New and deprecated Data Streamer

2021-06-08 00:00:00 +0000

An improved version of the Data Streamer is available. The update includes format and behaviour changes to help improve performance, stability and usability. In order to ensure a secure and seamless transition, the streams running on the old system are seamlessly migrated to the new system. Migrated streams are matching the previous format and your implementations will work like expected. Nevertheless, new features will be available only with the new system using the latest format, therefore we strongly recommend users to upgrade to the latest version by creating a new stream.

You can check if you are using a deprecated or migrated stream on the portal, the state gets indicated within the connection type name:

“Deprecated” - Your stream gets provided by the old system and will be migrated soon.
“Deprecated Format” - Your stream got migrated to the new system and you are using the deprecated format.

Streams on the new system providing the latest features are considered as the default state and therefore are not marked in any way.

Below is an overview of the differences between deprecated streams of the old system (“Deprecated”), the migrated to the new system (“Deprecated Format”) and the latest version.

Note: Most differences are shared between all destinations, destination-specific differences will be stated explicitly.

General Message Format Changes

This section is relevant for most destinations and for Event and Usage Data.

Timestamp Format

The format of timestamps e.g. for event->timestamp, sim->production_date, imsi->import_date.

Old streamer & migrated streamer (human readable with whitespace without time zone indication using UTC timezone):
```
"timestamp": "2020-11-10 08:59:19"
```
New streamer (ISO 8601 format using UTC timezone):
```
"timestamp": "2020-11-10T08:59:19Z"
```

Boolean Format

The handling of boolean values (e.g. alert field of Event Data)

Old streamer & migrated streamer (integer 0/1):
```
"alert": 0
```
New streamer (boolean true/false):
```
"alert": false
```

Null Values

Some values may be null in Events Data records. This is how they are handled by different Data Streamer versions.

Old streamer does not provide null root entities.

"endpoint": {
  "imei": "1234567890123456",
  "tags": null,
  ...
},
"description": "CloudConnect Transit Gateway breakout 132 is available",

Migrated streamer & new streamer provides also null root entities:

"endpoint": {
  "imei": "1234567890123456",
  "tags": null,
  ...
},
"description": "CloudConnect Transit Gateway breakout 132 is available",
"sim": null,
"user": null,

Field Name in `event_source`

The field of event_source has been renamed for consistency in the new system, but the old schema is kept for migrated streams.

Old streamer & migrated streamer:

"event_source": {
  "id": 0,
  "name": "Network"
}

New streamer:

"event_source": {
  "id": 0,
  "description": "Network"
}

Field Name in `traffic_type`

The field of traffic_type has been renamed for consistency on the new system, but the old schema is kept for migrated streams.

Old streamer & migrated streamer:

"traffic_type": {
  "id": 5,
  "name": "Data"
}

New streamer:

"traffic_type": {
  "id": 5,
  "description": "Data"
}

Character Encoding

Old system uses ASCII encoding and e.g. currency symbol is not shown correctly. New and migrated streams provide UTF-8 encoded data.

Old streamer:

"currency": {
  "id": 1,
  "symbol": " ",
  "code": "EUR"
}

Migrated streamer & new streamer:

"currency": {
  "id": 1,
  "symbol": "€",
  "code": "EUR"
}

Amazon Kinesis-Specific Changes

This section lists differences that are specific to Kinesis streams, apart from the already discussed changes in message format.

Error Behaviour

Old streamer
- does not indicate any errors if something is misconfigured
- stops working if a network error occurs
Migrated streamer & new streamer
- has retry mechanism based on KPL
- retrys transmiting data for up to 60 seconds, after which the stream will fail
- fails fast if first message cannot be delivered at all
- fails after some retries if destination stream becomes unavailable after previously successfull transmissions (e.g. deleted destination down)
- if an error occurs, the stream is switched to error state and transmitting stops (-> in that case, please fix configuration or destination and restart the stream)

Webhook / RestAPI-Specific Changes

This section lists specific changes to RestAPI which has been renamed to Webhook in the new system.

Headers

Migrated streamer & new streamer set a different User-Agent and two additional headers.

Old streamer
```
user-agent: TLD
```

Migrated streamer & new streamer

user-agent: DataStreamer/2.0
accept-encoding: gzip, deflate
accept: application/json

Error Behaviour

Old streamer fails on an undelivered message and indicates the HTTP Result Code. It does not respect redirects (HTTP Result Code 3xx) and retries constantly.
Migrated streamer & new streamer provide an improved error behaviour:
- Has retry policy with exponential backoff
- Fails fast if first message cannot be delivered successfully
- Failed data stream is put in error state and shows HTTP Result Code description or error message. In that case, please fix configuration or destination and restart the stream.

KeenIO-Specific Changes

This section lists specific changes to KeenIO data stream.

Setup of Data Stream

Old streamer delivers to fixed destination streams. With the new streamer, the ability to define the KeenIO collection names got introduced.
Migrated streamer will remain delivering data to the previous collection names defined for the old streamer.

Event Message Format

Old streamer & migrated streamer use a reduced event schema in comparison with RestAPI (e.g. does not deliver “imsi”). Migrated streamer contains additionally null root values.

"id": 1234,
"alert": 0,
"description": "PDP Context deleted.",
"detail": {
  ...
},
"endpoint": {                     
  "id": 1234,                    
  "name": "Test"                  
},                                
"event_severity": {
  ...
},
"event_source": {
  "id": 0,
  "name": "Network"
},
"event_type": {
  ...
},
"organisation": {
  ...
},
"sim": { 
  "id": 1234,                           
  "iccid": "8988303000000123456"                      
},                                  
"timestamp": "2020-11-10 08:59:19"

New streamer delivers the same schema as Webhook (therefore e.g. imsi, sim and endpoint have more information).

"id": 1234,
"alert": false,
"description": "PDP Context deleted.",
"detail": {
  ...
},
"endpoint": {
  "id": 1234,
  "imei": "1234",                                 // additional field
  "ip_address": "10.10.10.10",                    // additional field
  "name": "Test",
  "tags": null                                    // additional field
},
"event_severity": {
  ...
},
"event_source": {
  "id": 0,
  "description": "Network"
},
"event_type": {
  ...
},
"imsi": {                                         // additional field 
  "id": 1234,                                     // additional field
  "import_date": "2017-05-03T20:36:28Z",          // additional field
  "imsi": "123456789012345"                       // additional field
},                                                // additional field
"organisation": {
  ...
},
"sim": {
  "iccid": "8988303000000123456",
  "id": 1234,
  "msisdn": "123456789012345",                    // additional field
  "production_date": "2017-05-03T20:36:28Z"       // additional field
},
"timestamp": "2020-11-10T08:59:19Z",
"user": null

Message Format Usage Data

Old streamer & migrated streamer delivers reduced schema in comparison with RestAPI. Migrated streamer contains additionally null root values.

"cost": 0.0015,
"id": 1618839102882,
"operator": {
  ...
},
"organisation": {
  ...
},
"tariff": {
  ...
},
"traffic_type": {
  "id": 5,
  "name": "Data"
},
"endpoint": {
  "id": 1234,
  "balance": {                                    // old streamer: balance not present when not set 
    "amount": 15.0,                           
    "last_updated": "2020-05-29 15:38:36",    
    "expiry_date": "2025-04-06 09:00:00",         // old streamer: expiry_date not existing when not set
    "currency": {
      ...
    }                                       
  }                                             
},
"volume": {
  ...
},
"start_timestamp": "2021-04-19 13:31:42",
"keen": {                                         // old streamer: additional keen.timestamp that is the same as start_timestamp
  "timestamp": "2021-04-19 13:31:42"              // migrated / new streamer: not present, but automatically set by KeenIO upon receival
},                                              
"sim": {
  "id": 1234,
  "iccid": "8988303000000123456"
},
"currency": {
  ...
},
"end_timestamp": "2021-04-19 13:31:42"

New Data Streamer delivers the same schema as Webhook

"cost": 0.14776475,
"currency": {
  ...
},
"end_timestamp": "2021-02-22T13:05:45Z",
"endpoint": {
  "id": 1234,
  "imei": "1234567890123456",                     // additional field
  "ip_address": "10.10.10.10",                    // additional field
  "name": "Test",
  "balance": {
    "amount": 15.0,
    "last_updated": "2020-05-29T15:38:36Z",
    "expiry_date": "2025-04-06T09:00:00Z",
    "currency": {
      ...
    }
  }
},
"id": 1234,
"imsi": "123456789012345",
"imsi_id": 1234,
"operator": {
  ...
},
"organisation": {
  ...
},
"sim": {
  "iccid": "8988303000000123456",
  "id": 1234,
  "msisdn": "123456789012345",                    // additional field
  "production_date": "2017-05-03T20:36:28Z"       // additional field
},
"start_timestamp": "2021-02-22T13:04:41Z",
"tariff": {
  ...
},
"traffic_type": {
  "id": 5,
  "description": "Data"
},
"volume": {
  ...
}

Error Behaviour

Old streamer always shows Status 200 even if the data is not delivered successfully, e.g. because of wrong/invalid key. Status 500 is only shown if there is no response / no connection from KeenIO server.
Migrated streamer & new streamer applies the same retry strategy as Webhook and displays the returned error code.

Amazon S3-Specific Changes

Message Format Event Data

Old & migrated streamer CSV header: "id","timestamp","event_type_id","event_type_description","event_severity_id","event_severity_description","organisation_id","organisation_name","description","alert","event_source_id","event_source_name","endpoint_id","endpoint_name","endpoint_imei","endpoint_ip_address","endpoint_tags","sim_id","sim_iccid","msisdn_msisdn","sim_production_date","imsi_id","imsi_imsi","user_id","user_username","user_name"
New streamer CSV header (includes more fields): "id","timestamp","event_source_id","event_source_description","event_severity_id","event_severity_description","event_type_id","event_type_description","organisation_id","organisation_name","user_id","user_username","user_name","alert","description","endpoint_id","endpoint_name","endpoint_ip_address","endpoint_tags","endpoint_imei","sim_id","sim_iccid","sim_msisdn","sim_production_date","imsi_id","imsi_imsi","imsi_import_date","detail"

Message Format Usage Data

Old and migrated streamer CSV header: "id","event_start_timestamp","event_stop_timestamp","organisation_id","organisation_name","endpoint_id","sim_id","iccid","imsi","operator_id","operator_name","country_id","operator_country_name","traffic_type_id","traffic_type_description","volume","volume_tx","volume_rx","cost","currency_id","currency_code","currency_symbol","ratezone_tariff_id","ratezone_tariff_name","ratezone_id","ratezone_name","endpoint_name","endpoint_ip_address","endpoint_tags","endpoint_imei","msisdn_msisdn","sim_production_date","operator_mncs","country_mcc"
New streamer CSV header (includes more fields): "id","start_timestamp","end_timestamp","cost","volume_total","volume_rx","volume_tx","operator_id","operator_name","operator_mnc","operator_country_id","operator_country_mcc","operator_country_name","organisation_id","organisation_name","tariff_id","tariff_name","tariff_ratezone_id","tariff_ratezone_name","traffic_type_id","traffic_type_description","endpoint_id","endpoint_name","endpoint_ip_address","endpoint_tags","endpoint_imei","endpoint_balance_amount","endpoint_balance_last_updated","endpoint_balance_expiry_date","endpoint_balance_currency_id","endpoint_balance_currency_code","endpoint_balance_currency_symbol","imsi","imsi_id","sim_id","sim_iccid","sim_msisdn","sim_production_date","currency_id","currency_code","currency_symbol"