Advanced Data Projects May Be More Common Than Realized

Big data projects are increasingly finding their way into hybrid cloud environments, which suggests that advanced data projects may be significantly on the rise.

That is the take of Sumit Sarkar, chief data evangelist at Progress, who spoke to Information Management about what attendees at the recent Strata & Hadoop World conference in San Jose, CA, revealed about their present initiatives and challenges.

Information Management: What are the most common themes that you heard among conference attendees and how do those themes align with what you expected?

Sumit Sarkar: A common theme I heard repeatedly during Strata & Hadoop World was that big data platforms are either deployed on-premises or in the cloud, however, the data sources people want to access are often in a hybrid environment instead.

Our team predicted this would be the case in the future, but did not expect the notion to be true today as it indicates more advanced big data projects are in the works than we anticipated.

IM: What are the most common data challenges that attendees are facing?

SS: The biggest challenges I heard from attendees revolved around getting all the data they needed for a project at the right level of detail. Hosted cloud applications such as SaaS expose varying levels of details from their APIs, but many big data analytics projects are all about the low level details. Accessing a wide range of business systems across different platforms and RDBMS technologies presented a different set of challenges for the attendees I spoke with.

IM: What are the most surprising things that you heard from attendees?

SS: I was surprised at the interest from attendees in the management of data lakes in particular; and the supporting infrastructure for management across data preparation, ingestion, lineage, and quality, etc.