Overview

Role in the system

The defined, tested, monitored connections between applications, devices, databases, clouds and external agencies, so that independently built systems exchange data reliably and securely.

Systems may use different:

  • Data formats
  • Protocols
  • Authentication methods
  • Communication patterns
  • Update cycles
  • Error-handling mechanisms
  • Time references
  • Data models

The integration architecture must allow these systems to exchange information reliably and securely.

Controlled interface between two systemsData moves from a source system through validation, transformation, a queue and delivery to a destination system. Invalid data is reported, failed deliveries are retried with back-off and then held for investigation, reconciliation compares both ends, and logging and monitoring cover the whole interface.Authenticated, authorised and encrypted exchangeSource systemdata owner namedValidateformat and contentTransformmap to target modelQueuesurvives an outageDeliverconfirm receiptDestination systemdata owner namedinvalidRejection reportto data ownerretry with back-off, limitedafter limitHolding queueinvestigated, not discardedReconciliation: compare counts and values at both endsLogging and monitoring at both endsMessage rate, queue depth, age of oldest message, error count, latency; alerts go to the named interface owner
  • Data path
  • Power
  • Management path
  • Fault or alert
One controlled interface. Each interface validates, transforms, queues and delivers data, and defines what happens when data is invalid, delivery fails or the two ends disagree.

Integration architecture

Integration layers

Each layer has different failure behaviour: a device link drops intermittently, an agency interface changes without notice, a replication lag grows silently.

Integration may occur across:

  • Device-to-platform
  • Application-to-application
  • Database-to-database
  • Cloud-to-data-centre
  • Edge-to-cloud
  • User-to-application
  • External-agency interfaces
  • Monitoring-system interfaces

Integration methods

The method follows the interface’s timing and reliability needs: synchronous calls for lookups, queues and streams where delivery must survive an outage.

Methods may include:

  • Application programming interfaces
  • Message queues
  • File exchange
  • Database replication
  • Event streaming
  • Web services
  • Protocol gateways
  • Middleware
  • Enterprise service buses
  • Data pipelines

Interface discipline

Interface design

These twelve attributes form an interface specification. An interface with an undefined retry or reconciliation rule is an incident waiting for a trigger.

Each interface should define:

  • Source system
  • Destination system
  • Data owner
  • Data format
  • Frequency
  • Security
  • Error handling
  • Retry logic
  • Logging
  • Monitoring
  • Reconciliation
  • Change management

Integration testing

Testing covers abnormal conditions as deliberately as normal ones: bad, duplicated, late and missing data, and the loss of either end of the interface.

Integration testing should include:

  • Normal transactions
  • Invalid data
  • Duplicate data
  • Delayed data
  • Missing data
  • System unavailability
  • Network interruption
  • Retry behaviour
  • Reconciliation
  • Recovery

Discuss an ICT Requirement

Describe the requirement, the locations involved and where the project stands. We will respond with the technical questions that need answering first.

Discuss an ICT Requirement