Outline

• Operational Business Intelligence Systems
• Data Integration Patterns
• Quiz
• Process Integration

Introduction

• Last day, we said data integration is the general set of techniques that allow us to create a unified view of data from different sources.
• These techniques have evolved over the history of using databases for businesses.
• Originally, it was common that a different database might be used by each different department or organizational unit of a company, creating so-called data silos that couldn't inter-operate with each other (see left figure above).
• These would handle operational processes such as point-of-sale (PoS) storage of product purchases.
• Later, companies tried to combine these silos to support decision aspects of their business (i.e., business intelligence).
• This led to the development of data warehouses as we saw in the 1990s.
• Data warehouses allow business managers to make decisions that might be useful on the order of days, weeks, or months, but don't allow business to react to real-time changes in demand.
• They also tend to support only high level decisions, they are not suited to targeting individual consumers based on predicted demand.
• This led to the development of operational business intelligence systems - systems where operational and analytic processes are tightly integrated.
• We start today by describing and considering examples of such systems.

Operational Business Intelligence Systems

• Operational BI systems require low or zero latency so unexpected business-altering events and trends can be immediately detected and addressed.
• These systems move more and more of the batch processing involved in ETL to near real-time.
• Some examples of such systems are:
  • Executive dashboards that monitor KPIs (key performance indicators, such as production, stock prices, oil prices)
  • Business Process Monitoring or Business Activity Monitoring (BAM) that provide timely detection tools for anomalies or business opportunities.
• Big Data Analytics often tries to combine traditional operational BI data with internal and external data which can be more or less volatile such as clickstreams, server logs, sensor data, social media feeds, ... (i.e., streaming data.)
• Integrating data across these different kinds of resources at high speed often results in trade-offs between quality-of-service and quality-of-data.
• We next look at integration patterns that are used under different trade-off requirement scenarios.

Data Integration Patterns 1

• Data consolidation - capture data from multiple, heterogeneous source systems and integrate it into a single persistent store. This is the data warehouse/data mart approach that we have already considered. It involves an extraction, transformation and loading step, so tends to be higher latency, but have good data quality.
• Data federation - use a pull approach from underlying systems to integrate them on an demand basis. One example of this is Enterprise information integration (EII) where a virtual business view (with often a SQL interface) is presented based on underlying data sources. These data sources' internals are isolated from the outside world via wrappers. Queries performed on the business view are transformed into queries on the wrapper interfaces and executed on the underlying system and the results returned and combined back to the business view. This tends to be slower and not work well with complex queries.
• Data propagation - either synchronously or asynchronously propagate updates from a source to target system. This can either be done between two or more enterprise applications (enterprise application integration (EAI)) or between two or more data stores (enterprise data replication (EDR)). An example of EAI might be an order-handling application that triggers an invoicing application. This might be implemented using web services, RPC, or middleware. Data is propagated as part of the procedure call. With EDR, events in the source systems explicitly pertain to update events in the data store. These updates need to copied to a target store to allow for load balancing, availability, and recovery, or for real time loading of an analytics system.

Data Integration Patterns 2

• Changed Data Capture (CDC) - makes ETL event driven by detecting update events in the source data store and triggering an ETL process based on these updates. One can have more than one listener for different kinds of update events. More advanced systems support complex event processing where the triggering is not caused by single events but by a set of events or non-events in a time window. For example, a flurry of credit card purchases on a card may trigger a fraud detection application.
• Data Virtualization - aims to offer a unified data view for applications to retrieve and manipulate data without knowing necessarily where the data are stored or structured. This might sit as an abstraction layer above the data integration schemes described so far so that one can have timely access to combinations of federated and consolidated data. Often these systems offer caching, parallization of queries, and query optimization to improve response times.
• Service Oriented Architectures (SOAs) - support business processes by providing a set of loosely coupled software services. Such an integration suite might provide data service composition tools to allow these services to be combined and aggregated into a new, composite service. Such suites can provide "in-the-cloud computing". These services often involve hardware software and infrastructure provided on demand over a network. Possibilities include:
  • Software-as-a-service (SaaS) analytic application, cleansing, reporting etc hosted in the cloud.
  • Platform-as-a-service (PaaS) computing platform elements are hosted in the cloud and can be integrated with ones own applications (key-value stores in the cloud like Amazon S3).
  • Infrastructure-as-a-service (IaaS) hardware infrastructure offered as virtual machines in the cloud.
  • Data-as-a-service (DaaS) data services hosted in the cloud such data quality monitoring, cloud based data integration, etc. allowing one to integrate one's own data with data from external providers

Quiz

Which of the following is true?

1. Dependent attributes are linked to a fact table by foreign key relationships in the star schema.
2. To return the results that might be in a formal data cube, we might use a GROUP BY CUBE clause in SQL.
3. Operational data stores are small scale data warehouses designed to meet the information needs of a small group of end users such as a department or business unit.

Data Services, Data Flows, and Process Integration

• Process integration is the integration and harmonization of business processes in an organization.
• A business process is defined to be a set of activities with a certain order that must be executed to reach a certain organizational goal.
• For example, a loan approval process might consist of tasks such as filing a loan application, calculating a credit score, drafting a loan offer, signing the contract, etc.
• These steps can typically be arranged as a directed acyclic graph which says from where-to-where data flow goes and what data is involved.
• From an implementation perspective in an information system, one often views task coordination/triggering and task execution as intertwined in the same software code.
• From a service-oriented perspective, one tends to separate services whose job is task coordination from services that perform the actual task execution and from services that provide access to the necessary data.

Business Process Integration

• The directed acyclic nature of business process is often modeled using flow chart like tools such as: business process model and notation (BPMN), Yet Another Workflow Language (YAWL), Unified Modeling Language (UML) Activity Diagrams, Event-driven Process Chain (EPC) diagrams, and so on.
• Above we can see one such diagram (BPMN) for the loan approval process.
• Such a process chart can be used as-is by workers in an ad hoc fashion or it can be used to drive steps of an information system, aiding in the automation of the process at hand.
• Such an automation application is called a process engine.
• These systems are coded using a sequence of declarative steps often in a (Prolog-like) language such as WS-BPEL (Business Process Execution Language).
• There are a number of design patterns used to develop processes for such systems which we will describe next day.

Approaches to Process Management

• Process execution languages such as WS-BPEL often distinguish between two kinds of dependencies that relate one activity in a process has on another:
  • A sequence dependency - execution of Service A must complete before execution of Service B. For example, a loan proposal can only be made after a positive credit score has been calculated.
  • A data dependency - says that Service B requires data produced by Service A. For example, an interest rate of a loan proposal that depends on a credit score calculated during the credit check.
• Two common patterns for managing sequence and data dependencies of businesses processes are:
  • The orchestration pattern - (left above, used by WS-BPEL) Assume a single centralized executable business process called the orchestrator that coordinates the interaction among different services and subprocesses. The orchestrator is responsible for invoking, combining, and receiving updates of process and services.
  • The choreography pattern - the participants in the process itself coordinate their collaboration.

Combining Data and Process Integration

• There are two common ways that a data dependency between service A and service B can be resolved:
  • Provide a data flow from A to B to make sure the necessary data are passed between the services at the level of the business process..
  • Have Service A persist data into a data store which is also accessible through one of our data integration techniques by Service B.
• To analyze data flow at the process and data integration level in a unified way, we consider three types of services:
  • Workflow Services (might correspond to a process engine) coordinate the control flow and data flow of a business process by triggering its respective tasks in line with the sequence constraints in the process model, and according to an orchestration or choreography pattern. Steps may be executed by a machine or human and results returned to feed into rest of the process.
  • Activity Services (might correspond to a web service) perform one task in a business process and return a result which can be used by the workflow process. For example, credit card validation
  • Data Services provide unified access (say via federation, consolidation, virtualization, etc) to the business data in underlying data stores at some kind of level of quality of service as defined by response time, completeness, consistency, etc.

Data-related Infrastructure Services

• Most service-oriented architecture data integration suites (used to build process and data integration architecture like we've been talking) provide different data-related infrastructure services that support the exploitation and management of data services.
• These are often related to data quality and data governance.
• Some example services include:
  • Data profiling services - providing automated support for assessing and understanding content, quality, and structure of enterprise data.
  • Data cleansing services - ensuring the validity and consistency of data For example, name and address cleansing
  • Data enrichment services - enhance the data by exploiting external data sources.
  • Data transformation services - tool for transforming data to match the target application's requirements
  • Data event services - tools for monitoring data for state changes and rules, raising events that can be acted upon by other services.
  • Metadata services - support the storage, integration and exploitation of diverse types of metadata