Latest News

New survey finds majority of Data Engineers are hampered by constant infrastructure challenges

While data-driven decision making is the number-one indicator of success for today’s businesses, a new global study conducted by Dimensional Research for Fivetran, the leading automated data integration provider, shows organisations largely unable to leverage the ever-increasing amount of data they have on hand.

According to the survey of 543 data professionals across five continents, 44 percent of respondents say that key data is not yet usable for decision-making and 68 percent say additional business insights could be extracted from existing data if they only had more time. This is largely attributed to inefficiencies faced by data engineers, who are responsible for transforming data throughout an organisation into a format that can be easily analysed.

Key findings include:

  • Nearly all participants (98 percent) have problems building pipelines to access their data.
  • 98 percent of participants also say their pipelines break, with 51 percent stating it happens more frequently than once a month.
  • Respondents who reported pipelines breaking daily cite source availability and data schema changes as the top reasons.

Eng-Survey-Stat4.png

As a result, business suffers with data engineers spending too much time fixing and maintaining pipelines, rather than focusing on higher-value tasks that generate a positive impact on the business. 66 percent report reduced operational efficiency and 59 percent report delayed decisions or lost opportunities due to broken pipelines.

“Data engineers are the forces behind revenue-impacting business decisions, yet they are still burdened by the limitations of brittle, hard-to-maintain data pipelines. Companies should be aware of the challenges these valuable employees face so they can deliver reliable data more easily to business stakeholders,” said George Fraser, CEO of Fivetran. “Fivetran provides reliable access to data through fully managed data pipelines and SQL transformations – with 99.9 percent guaranteed uptime. This allows companies to put data to work and make decisions faster, instead of spending valuable resources on building and maintaining pipelines themselves.”

Eng-Survey-Stat6.png

For large enterprises and small-to-mid size organisations alike, having immediate and reliable access to data is critical for decision-making. The study indicates that pipeline challenges for data engineers continue to strain the process, and businesses lose out on valuable insights as a result.

Additional findings from the survey include:

  • 79 percent of companies plan to hire more data engineers within the next year
  • 72 percent of organisations move source data daily or more frequently
  • 59 percent of companies use 11 or more data sources
  • 91 percent say source availability problems are the reason why pipelines break daily
  • 55 percent say it takes longer than one business day to repair pipelines
  • 86 percent use multiple solutions to build new data pipelines
  • 41 percent say a lack of data strategy (policies, processes, standardised technology) is the top data-related challenge their organisation faces