Data Integration Technology Selection

On the surface data integrations seems easy but selecting the right technology to match the right outcome can be anything but

The purpose of this document is to help guide and inform my IT teams to use the correct integration platform for their use cases. Note that this is only a guide, as there can be multiple options based on the integration activities.

As you think about your data and integration requirements, you can think about it strategically or tactically — the value you provide is correlated. The Data Integration Maturity Matrix diagram below visually helps to show where your integration activity contributes to the overall maturity of the organization. Then use the Technology Decision Matrix in this document to help you determine which IT-supported technology fits your needs. Let’s review each integration activity on the maturity matrix and discuss the business and technical drivers, technical considerations, and use cases. Each has a recommended IT technology associated with it.

Data Virtualization

Data virtualization is an approach to data management that allows an application to retrieve and manipulate data without requiring technical details about the data, such as how it is formatted or where it is physically located.

Business Drivers

• Integrated view of information in one interface
• User productivity and reduced errors

Technical Drivers

• UI-level integration
• Drive consistency without replicating or syncing data
• Create a unified view of data via SFDC OData XDS Connector

Considerations

• Must expose data sources as OData API feeds
• Limited to UI-level integration — no data transformation or orchestration of data for more strategic value
• Point-to-point integration; tightly coupled to the backend data source
• Salesforce Connect only supports read-only from external data sources (FY15Q4)

Use Case Examples

• Mobile phones and Web browsers access a custom application’s data
• Application business logic access data exposed in the cloud
• Salesforce applications wanting to access legacy without data movement
• Different BI tools to access diverse data sources

Recommend IT Technology

• MuleSoft
• JitterBit

Data Load

Copying data from one electronic file or database into another. Data loading implies converting from one format into another; for example, from one type of production database into a decision support database from a different vendor. Commonly used in data warehouse scenarios.

Business Drivers

• One time system or application cutover
• Loading cleansed Lead, Contact and Customer data

Technical Drivers

• Migrating large amounts of data at once (bulk load)
• Support for complex data transformations
• Address specific data need for narrow use cases

Considerations

• May require multiple data loading attempts to complete objective
• Latency in data movement must be considered in requirements
• May require data loading logic to obtain optimal performance
• Tightly coupled architecture; Increased cost to maintain changes

Use Case Examples

• Loading bulk data files into a data warehouse
• Migrating or copying data in tables from one database to another database
• Replicating bulk data files from a server or location and loading them into a database
• Moving data into a backup, disaster recovery, or failover database
• Loading bulk data files or tables into a cloud-based database

Recommend IT Technology

• MuleSoft
• Informatica
• JitterBit
• Salesforce Import Wizard

Data Synchronization

The process of establishing consistency among data from a source to a target data storage and vice versa and the continuous harmonization of the data over time.

Business Drivers

• Build and maintain unified Customer databases
• Provide consistent customer experiences across systems
• Improve business decisions and process performance

Technical Drivers

• Data consistency from one or more systems or applications
• Real-time or scheduled data synchronization
• Maintain data integrity

Considerations

• Frequency or latency needs over time; i.e. real-time vs. scheduled
• Increased data volume over time can impact business objective
• Implementation can vary data reconciling needs (small vs. large batches)
• Data profiling activities recommended prior to implementation
• Handling data conflicts when they occur and resolving taking into account record update precedence or “record owner”

Use Case Examples

• Scheduled data warehousing for operational business intelligence and analytics
• Continued data migration for consolidation and synchronization among multiple systems

Recommend IT Technology

• MuleSoft
• Informatica
• JitterBit

Process Automation

The strategy a business uses to automate processes in order to contain costs. It consists of integrating applications, restructuring labor resources, and using software applications throughout the organization.

Business Drivers

• Automate business processes that span multiple departments
• Drive richer customer experiences through real-time intelligence
• Drive lower operational costs for department and/or company
• Reduce significant compliance deficiencies

Technical Drivers

• Data and interactions fragmented across multiple departments
• Create and orchestrate processes across application silos to meet internal SLAs

Considerations

• Accessibility to data and applications via APIs
• Data ownership, location, and availability of data
• Real-time business process automation

Use Case Examples

• Automation of new hire onboarding activities
• Real-time automation of quote to cash business process
• Automation of a complex and critical outage incident management process

Recommend IT Technology

• MuleSoft
• Informatica

Shared Services

A collection of services that communicate with each other. The communication can involve either simple data passing or it could involve two or more services coordinating some activity. A means of connecting services to one another is needed. Faster pricing changes, new applications, bring products to market faster, better customer services, etc.

Business Drivers

• Deliver products and projects more rapidly to the market
• Discover opportunities and innovate more quickly
• Drive lower operational costs for department and/or company

Technical Drivers

• Code reuse and application decoupling to promote agility
• Easier to maintain large integration projects
• Standardized data definitions and format

Considerations

• Requires C-level executive support to adopt a complete SOA strategy
• A central/core service offering is required to span departments as an internal service and integration provider
• Upfront cost and effort to realize long-term ROI

Use Case Examples

• Guaranteeing the delivery of data events such as a customer order
• Loosely coupled from source and target systems reducing costs for future system modifications
• Promotes code reuse when new systems require access to existing data

Recommend IT Technology

• MuleSoft

APIs

An Application Programming Interface (API) specifies a software component in terms of its operations, its inputs and outputs, and underlying types. Its main purpose is to define a set of functionalities that are independent of their respective implementation, allowing both definition and implementation to vary without compromising each other. Leveraging data better within the enterprise to drive efficiency, collaboration, etc. (private API). Capitalizing on data to drive new channels and revenue streams (public API).

Business Drivers

• Transform business to embrace mobility and connected devices
• Monetize digital assets and create new revenue channels
• Drive lower operational costs for department and/or company

Technical Drivers

• Makes data in enterprise and custom applications readily available
• Ability to secure data, track, and audit usage
• Decouples consumers from source applications
• Drives awareness for data definitions and data quality

Considerations

• More API users will lower overall costs and increase ROI
• Requires technical resources to implement
• Design with an audience in mind (audience-based design)
• Think about the entire API lifecycle, not just the initial implementation

Use Case Examples

• Real-time access to data for mobile and web applications
• Automating manual business processes through orchestrating one or more APIs

Recommend IT Technology

• MuleSoft

Technology Decision Matrix

Choosing the right integration technology for a project can be an overwhelming task for many, so I created the matrix below to help guide the user to the right technology for the intended outcome. It should be noted that the matrix should be used in conjunction with input from our Enterprise Architecture Review Board (EARB) regarding appropriate technology since specific use cases will play a key role in determining the appropriate technology.

Note that this framework can be adjusted to meet other capabilities and/or technology that your organization uses. Please drop a comment and let me know if you find this helpful.