For more than 20 years, data engineering has been overshadowed by other initiatives like business
intelligence, data integration, data quality, master data management (MDM) and extract, transform and
load (ETL). However, with the rise of technologies like Hadoop, Spark, Internet of Things (IoT), cloud
computing, stream processing, data science, machine learning (ML) and artificial intelligence (AI), data has
become more readily available to users outside of IT.More recently, the rise of data science and AI/ML practices have increased the demand for data
engineering to make data available in a readily consumable form. According to a survey of data
professionals, respondents indicated that they spend close to 40% of their time cleaning and transforming
data.2