Chapter 1: Basics of Data Engineering

As a data scientist looking to transition into data engineering, you’ve likely encountered the term “data

engineering” quite a bit. It’s a hot field, and for good reason — data engineers build the foundation that data

science and analytics are built upon. But what exactly does a data engineer do?

The first chapter dives right in, defining data engineering and exploring its evolution.

Defining Data Engineering

The chapter acknowledges the confusion surrounding data engineering. There are many definitions floating

around, but here’s the key takeaway:

Data engineering is the process of developing, implementing, and maintaining systems that transform raw data

into high-quality, usable information for data scientists, analysts, and other consumers. It’s the bridge between

raw data and actionable insights.

Think of data engineering as an intersection of various disciplines: security, data management, data operations

(DataOps), data architecture, orchestration, and software engineering. Data engineers manage the entire data

lifecycle, from acquiring data from various sources to preparing it for analysis and machine learning.

Lifecycle of Data Engineers

This lifecycle focuses on the data itself and the ultimate goals it serves, rather than getting bogged down in

specific technologies.

There are five key stages in this lifecycle:

1. Generation: This is where the data originates from.

2. Storage: Here, the data is housed in a secure and accessible location.

3. Ingestion: The data is then brought into the system for processing.

4. Transformation: The raw data is cleaned, transformed, and prepared for analysis.

5. Serving: Finally, the transformed data is delivered to those who need it, such as data scientists and analysts,

for their use cases.

Several underlying principles, essential throughout the lifecycle, are also mentioned. These include security, data

management, DataOps practices, data architecture, orchestration, and software engineering.

A Brief History of Data Engineering

Here are some key takeaways, to evaluate data engineering:

 The early days (1980s-2000s) saw the rise of data warehousing and Business Intelligence (BI) engineers.

 The early 2000s witnessed the birth of “big data” due to the explosion of data and advancements in

distributed computation and storage.

 Public cloud platforms like AWS emerged, offering scalable and cost-effective data storage and processing.

 Open source big data tools like Hadoop became popular, but managing them required significant effort.

The Present and Future of Data Engineering

The focus has shifted towards simplification and abstraction of data tools. Data engineers are now more
concerned with managing the entire data lifecycle, including security, data governance, and compliance. This has
led to the rise of the “data lifecycle engineer.”

Data engineering and data science are complementary fields. Data engineers provide the clean data that data

scientists use to build models and extract insights.

Becoming a Data Engineer: Skills and Background

This section explores the background and skills necessary for a data engineer. Data engineering is a relatively

new field, so there’s no formal training path. People from various backgrounds enter this field, and self-study is

crucial.

Moving into Data Engineering

The transition to data engineering is smoother from adjacent fields like software engineering, database

administration, or data science. These fields provide relevant technical skills and context for data engineering

problems.
Essential Knowledge for Data Engineers

Data engineers should possess knowledge of both data and technology. Regarding data, this means understanding

data management best practices. On the technology side, they should be familiar with various data tools and their

trade-offs. Additionally, they should understand software engineering principles, DataOps, and data architecture.

Data Engineers and the Bigger Picture

Beyond technical skills, data engineers should understand the needs of data users (analysts and scientists) and the

broader impact of data within the organization. They should be able to communicate effectively with both

technical and non-technical audiences.

Data Maturity in the Context of Data Engineering

Data maturity refers to the evolution of an organization’s ability to harness data effectively across its various
functions.
This concept does not strictly depend on the age or financial scale of a company. Instead, it focuses on how
data is utilized to gain a competitive edge.
For data engineers, understanding the levels of data maturity is crucial as it directly impacts their
responsibilities, workflow, and career development.

Simplified Data Maturity Model for Data Engineering

For practical purposes, we will explore a simplified data maturity model consisting of three stages:

1. Starting with Data

2. Scaling with Data

3. Leading with Data

Each stage represents a phase in the organization’s data utilization and sophistication, influencing the role
and focus of data engineers.
Stage 1: Starting with Data

At this initial stage, organizations are beginning to explore the potential of data. Characteristics of this stage

include:

 Undefined Goals: The organization might not have clear data-related objectives.

 Nascent Infrastructure: Data architecture and infrastructure are in early development phases.

 Low Adoption: Usage of data within the company is minimal, with most data requests being ad hoc.

 Role of Data Engineers: Data engineers act as generalists, handling multiple roles including that of data

scientists and software engineers. Their primary focus is to establish a robust data foundation and achieve

quick, impactful wins despite potential technical debt.

Key Responsibilities:

 Gain executive buy-in for data initiatives.

 Design and implement a suitable data architecture.

 Identify and prepare data that aligns with business goals.

 Avoid unnecessary complexity and use off-the-shelf solutions wherever possible.

As a data engineer in the initial stages of data maturity, my advice is to emphasize speed and flexibility.
Focus on quickly building and deploying functional systems to gather insights and iterate based on real-
world feedback.

Avoid striving for perfection and prolonged phases of development; instead, learn from what you have
deployed and continuously improve. This approach ensures you move forward and adapt effectively,
crucial for growth and success in early-stage data engineering.
Stage 2: Scaling with Data

As the company matures in its data journey, the approach transitions from ad hoc to formalized data practices.

Characteristics of this stage include:

 Formal Data Practices: The organization adopts structured data handling and processing methods.

 Specialization of Roles: Data engineers begin to specialize in specific aspects of data engineering.

 Integration of DevOps and DataOps: These practices become crucial in managing data workflows

efficiently.

Key Responsibilities:

 Establish scalable and robust data architectures.

 Implement systems that support machine learning.

 Continue to refine and optimize data practices to prevent overengineering and maintain focus on delivering

value.

Stage 3: Leading with Data

In the final stage, the organization is truly data-driven, characterized by advanced data integration and analytics

capabilities. Characteristics of this stage include:

 Automated Data Systems: Automated pipelines and systems facilitate self-service analytics and machine

learning.

 Deep Specialization: Data engineering roles become highly specialized.

 Strategic Data Utilization: Data is extensively leveraged as a strategic asset.

Key Responsibilities:

 Automate data integration and usage.

 Focus on data governance, quality, and management.

 Develop tools that enhance data accessibility and understanding across the organization.

Common Challenges:

 Complacency and Maintenance: Organizations must continually invest in maintaining and upgrading their

data capabilities to avoid regression.

 Avoiding Technology Distractions: It’s crucial to focus on technologies that offer real, measurable business

value rather than pursuing “hobby projects.”

Business Responsibilities

 Communicate with technical and non-technical people.

 Understand how to scope and gather business and product requirements.

 Grasp the cultural foundations of Agile, DevOps, and DataOps.

 Control costs and optimize for time to value, total cost of ownership, and opportunity cost.

 Continuously learn and stay updated with the evolving data landscape.

Technical Responsibilities

 Design architectures for optimal performance and cost, using pre-built or custom components. These

architectures should serve the stages of the data engineering lifecycle (generation, storage, ingestion,
 transformation, serving) while considering security, data management, DataOps, data architecture, and

software engineering principles.

Data and Technology Skills

Programming Languages:

Primary Languages:

 SQL: Most common interface for data storage and retrieval.

 Python: A versatile language for data engineering and data science, often used for data manipulation and

interacting with data tools.

 Java or Scala (JVM languages): Commonly used in Apache open-source data projects like Spark.

 Bash: Command-line interface for Linux systems, essential for scripting and OS operations.

 Secondary Languages: Exposure to languages like R or Julia may be beneficial depending on the role.

Data engineers need a blend of business acumen, communication skills, and technical expertise in data

management, software engineering, and specific programming languages. This book offers a roadmap to acquire

these skills and knowledge and succeed in the data engineering field.

Data Engineering Roles and Responsibilities

This section dives into various data engineering roles and how data engineers interact with other technical and

non-technical personnel within an organization.

The Data Engineering Continuum

Data engineering isn’t a one-size-fits-all field. There’s a spectrum of data engineering roles, with some focusing

on using existing tools (Type A) and others on building custom systems (Type B). This distinction is similar to

how data science can be divided into Type A (analysis) and Type B (building) data scientists.

Data Engineers and Their Interactions

Data engineers collaborate with various technical and non-technical teams throughout the organization. Here’s a

breakdown of these interactions:

 Internal-Facing vs. External-Facing Engineers:

 Internal-facing engineers deal with data pipelines and data warehouses for business dashboards, reports, and

internal data science projects.

 External-facing engineers design systems that collect, store, and process data from external applications like

social media or IoT devices.

Technical Roles Data Engineers Interact With:

 Data Architects: Design the overall data architecture and blueprint for data management.

 Software Engineers: Build the applications that generate the data data engineers will process.

 DevOps/SRE Engineers: Maintain operational systems and produce data through monitoring.

 Data Scientists: Develop models that use the data provided by data engineers.

 Data Analysts: Analyze data to understand business trends and performance.

 ML Engineers: Develop and maintain ML infrastructure and processes.

 AI Researchers: Research new and advanced ML techniques.

Data engineers play a central role in data management and interact with various stakeholders across the

organization. Understanding these interactions is crucial for a successful data engineering career.

This chapter explores the role of data engineers beyond technical aspects and emphasizes their importance in

business leadership.

Data Engineers and C-Suite

Data engineers collaborate with C-suite executives who increasingly view data as a strategic asset. They help

CEOs understand the potential of data and maintain a data inventory for the organization.

 Chief Executive Officer (CEO): Defines the data vision and collaborates with data engineers to understand

data capabilities.

 Chief Information Officer (CIO): Oversees IT and works with data engineers on data initiatives and

architectural decisions.

 Chief Technology Officer (CTO): Owns the technology strategy for external applications (data sources for

engineers).
 Chief Data Officer (CDO): Manages the company’s data strategy and assets, often working with data

engineers.

 Chief Analytics Officer (CAO): Focuses on analytics, strategy, and data-driven decision making (may

oversee data science).

 Chief Algorithms Officer (CAO-2): Highly technical role leading data science and ML initiatives.

Data Engineers and Project/Product Management

Data engineers collaborate with project managers who prioritize tasks and ensure projects stay on track. They

also work with product managers who oversee data product development.

Data Engineers and Other Management Roles

Data engineers may interact with various managers depending on the company structure. They may function as a

centralized service team or be assigned to specific projects/products.

Conclusion

Data engineers are not isolated code hackers. They need to understand the problems they solve, the tools they

use, and the people they work with. This chapter introduced data engineering, data maturity levels, data engineer

types, and their interactions within an organization.

Understanding and navigating through the stages of data maturity is essential for data engineers aiming to
effectively contribute to their organizations’ data-driven ambitions.
By recognizing the characteristics and demands of each stage, data engineers can better align their strategies
and actions with the organization’s overall data goals.
This strategic alignment not only accelerates personal career growth but also enhances the organization’s
competitive position in the industry.