Data Integration: Choosing the Best Approach

Jul 22, 2016
6 Min Read
Data Integration: Choosing the Best Approach

The total volume of enterprise data stored in marketing and cloud-based systems is expected to grow by 50 percent each year to around 40 zettabytes, or 40 billion terabytes, by 2020, according to IDC.

With the cloud only getting bigger and more successful, data integration is the only way your company can use its own data to create actionable insights.  Carefully choosing an approach to data integration is essential—as it must allow you to easily combine diverse data types, independent of the location or application that generated it.

One of the most important trends in this area is “self-service integration,” which involves individuals in an emerging role that Gartner calls citizen integrators—those who perform do-it-yourself integration tasks without support from IT departments or expert developers.  There are three general approaches to self-service integration.

  1. Manual Integration

Manual data entry will introduce unwanted errors and ultimately isn’t real integration of data across sources but a clumsy workaround. IDC has shown that at least 90 percent of data in the cloud going forward will be unstructured, so the task of hand-copying data into different locations will become more complex.   While standard import/export functionality (i.e. supporting XLSX, CSV, TSV, XML and other formats), provided by most applications, reduces human error it's still a manual process subject to vacation schedules, sick days, or memory slips.  Avoid manual data entry if at all possible.

  1. Pre-built Connectors

Some standalone applications and application Platform-as-a-Service (aPaaS) offer pre-built integrations or connectors that automate the movement of larger data sets. The upside is that you don't need to code to build an integration, and the downside is that these tend to be limited in use and tend to work with specific applications.

However, aPaaS has an edge over standalone Software as a Service (SaaS) applications because apps built on one platform can be integrated more easily and through a greater choice of options (i.e. table-to-table import, report link, or cross-app relationships). Also, you have more choices to leverage any data that you bring into a platform than you do with a standalone app. You can create reports/charts/dashboards, connect with other data available in other apps built on the platform, and share with the right people, at the right level of detail, at the right time— resulting in a richer, more complete picture of the business.

An example is QuickBase, which offers both pre-built integrations (such as QuickBase Sync which integrates QuickBase with apps like Salesforce or Dropbox) and custom add-ons (such as those developed by QuickBase Solution Providers who are third-party consultants with expertise in customization for specific business needs and use cases).

  1. Self-Service Integration Tools

According to Gartner the promise of emerging citizen integrator tools (aka self-service integration tools) is to increase business users' efficiency, agility and ability to innovate via do-it-yourself automation of integration tasks.  Such tools are beginning to support the creation of increasingly complex and sophisticated integration flows- through a simple user interface, without coding.

Self-service integration tools are designed specifically for accessibility and ease of use to enable business experts with no programming skills to build integration flows rapidly.  Such tools (some of the most innovative are Workato and Zapier) often partner with aPaaS like QuickBase, typically require no coding (they rely on drag and drop), and enable task automation and data integration across more than a hundred applications.

Many of these platforms are beginning to integrate sensors and physical devices with an API (application programming interface). Such integration will become more important as the Internet of Things becomes more and more important to development. These products often have enthusiastic community that shares their app integrations— Workato calls these “recipes”.

What’s Next?

The citizen integrator phenomenon is the most promising trend for data and application integration but is still in its early stages. Innovative integration providers like Zapier and Workato simplify the complex integration process. Self-service integration tools like Workato enable the most complex and sophisticated intentions, connecting both cloud-based and on-premises applications, with more understanding of how to work with established data governance and security infrastructures.

No matter what approach you take, there will still be unmet data integration needs. Integration across an expanding universe of applications and spreadsheets (what Gartner calls “spreadsheet proliferation”) will continue to be a challenge. While custom integration enabled by APIs will continue to play an important role, organizations will be building an increasing number of integration flows with self-service tools.

Recomended Posts