Quick Review: Data Modeling, Data Lineage, and Schema Evolution

Purpose of This Lesson

This lesson provides a focused review of three data engineering concepts relevant to the AWS Certified Data Engineer Associate: dimensional data modeling, data lineage, and schema evolution. The emphasis is on recognizing the concepts, understanding their operational value, and selecting the appropriate AWS or related tool in a scenario.

Key Concepts

Star Schema

A star schema organizes data around a central fact table connected to multiple dimension tables.

• The fact table records measurable business events, such as a student enrolling in a course.
• Dimension tables hold descriptive information about the entities involved in those events.
• Primary and foreign keys connect the fact table to its dimensions.
• The structure is called a star schema because the fact table is at the center and the dimension tables branch outward.

For an e-learning platform, an enrollment fact table might contain a course ID, student ID, payment ID, and order timestamp. Related dimensions could provide course details, student information, and payment attributes.

The model reduces duplication in the fact table by storing descriptive details in separate dimension tables. An entity relationship diagram (ERD) can be used to represent the entities, attributes, and relationships visually.

Data Lineage

Data lineage is a visual or documented representation of how data flows and changes from its source to its final destination. It records movement, transformations, and consumption across a data pipeline.

Lineage is useful for:

• Tracing an incorrect value back to its source or transformation step.
• Troubleshooting extraction, transformation, and loading pipelines.
• Demonstrating how sensitive or regulated data was handled.
• Helping new engineers understand the path data takes through the system.

In the AWS example discussed, AWS Glue ingests data from Amazon S3, catalogs and transforms it, and a Spline Agent attached to Glue captures the processing history. Spline exposes a Lineage API, which can be consumed by tools or by an AWS Lambda function. Lambda can write lineage information to Amazon Neptune, whose graph-database structure is suitable for representing relationships among data sources, transformations, and destinations.

The lesson also identifies SageMaker Lineage as another lineage capability encountered in AWS contexts. Spline Agent, however, is the third-party tool used in the specific Glue example.

Schema Evolution

Schema evolution is the ability to change a dataset’s schema over time without disrupting existing processes or systems.

Schema evolution can involve adding, removing, or modifying columns and fields. A well-managed evolution strategy supports changing business requirements while preserving compatibility with older records and existing consumers.

This flexibility is associated with data lakes because the data format does not necessarily have to be rigidly fixed at the time data is first stored. It reduces the need for disruptive migrations whenever requirements change.

The AWS Glue Schema Registry provides schema discovery, compatibility, validation, and registration. It manages different schema versions and can help ensure backward compatibility with older versions of data.

Data Modeling, Lineage, and Schema Evolution in Practice

These concepts address different problems in a data platform:

Concern	Main question	Relevant concept or tool
Organizing analytical data	How should events and descriptive entities be structured?	Fact tables, dimension tables, star schema, ERD
Understanding data movement	Where did this value come from, and what happened to it?	Data lineage, Spline Agent, Lineage API, SageMaker Lineage
Changing data structure safely	How can the schema change without breaking existing consumers?	Schema evolution, Glue Schema Registry

A data engineer may use all three together. For example, an analytical data platform can use a star schema for reporting, lineage records to document how source data became reporting data, and schema-versioning controls to introduce new fields safely.

Exam- or Assessment-Relevant Takeaways

• A star schema has a central fact table and surrounding dimension tables.
• Fact tables generally represent events or measurements; dimension tables provide descriptive context.
• Primary and foreign keys connect facts to dimensions.
• An ERD visually represents entities and their relationships.
• Data lineage traces data from source through transformations to its final destination.
• Lineage supports troubleshooting, compliance documentation, and onboarding.
• In the described AWS example, Spline Agent is attached to Apache Spark processing in AWS Glue to capture lineage.
• A Spline Lineage API can expose captured lineage for downstream tools or processing.
• Amazon Neptune is a reasonable storage target for lineage information because lineage naturally has a graph structure.
• Schema evolution means changing a schema without disrupting existing systems or processes.
• AWS Glue Schema Registry supports schema discovery, validation, registration, compatibility, and version management.
• Do not confuse Spline Agent in the specific Glue example with SageMaker Lineage, which is another lineage capability mentioned in the lesson.

Tool / Feature Decision Guide

Choose a Star Schema When

The problem is analytical organization around business events and their descriptive attributes. Put event-level records in a fact table and reusable descriptive details in related dimensions.

Choose an ERD When

You need to communicate or inspect the entities, attributes, and key-based relationships in a data model. An ERD is a modeling representation, not a lineage or schema-versioning service.

Use Data Lineage When

The main requirement is to understand where data came from, which transformations it passed through, and where it was consumed. Lineage is especially valuable when investigating incorrect transformations or documenting regulated data handling.

Use Spline Agent in the Described Glue Pattern When

Apache Spark processing in AWS Glue needs to emit lineage information. Spline captures the processing activity and makes it available through its Lineage API.

Use Amazon Neptune for Lineage Storage When

The implementation needs to store and query relationships among sources, transformations, and destinations. Its graph-database model fits the graph-like structure of lineage.

Use AWS Glue Schema Registry When

The main problem is managing changing schemas and maintaining compatibility across schema versions. It provides schema discovery, validation, registration, compatibility handling, and version management.

Common Traps / Misconceptions

• A fact table is not simply a general-purpose table; in a star schema it records business events or measurements and links to descriptive dimensions.
• Dimension tables do not replace the fact table. They provide context for the keys stored in facts.
• An ERD describes data structure and relationships; it does not record the historical path or transformations of data.
• Data lineage is not the same as schema evolution. Lineage tracks data flow, while schema evolution manages structural change over time.
• Schema evolution does not mean changing fields without controls. Compatibility and validation are important when older records or consumers still exist.
• Spline Agent is not the same product as SageMaker Lineage. The lecture’s Glue example specifically uses Spline Agent with a Lineage API.
• Amazon Neptune is used in the example to store lineage information; it is not presented as the service that captures the Glue transformation itself.
• Schema evolution is not limited to adding fields. It can also involve removing or modifying columns and fields, provided existing processes remain supported.

Real-World Engineer / Analyst Notes

• Treat lineage as operational documentation, not merely as a diagram. It should help answer which source and transformation produced a particular output.
• When a pipeline produces incorrect data, lineage narrows the investigation to the relevant source, transformation, or handoff.
• Capture schema versions and compatibility expectations before introducing new producers or consumers.
• A graph representation is useful when the important question is relationship-based—for example, which downstream datasets depend on a particular source.
• Keep the modeling decision separate from the governance decision: a star schema organizes analytical data, while lineage documents its journey and schema management controls its structural changes.

Quick Reference Summary

• Fact table: Stores event or measurement records.
• Dimension table: Stores descriptive attributes related to facts.
• Star schema: Central fact table connected to dimensions by keys.
• ERD: Visual representation of entities and relationships.
• Data lineage: Trace of data from source through transformations to destination.
• Spline Agent: Third-party lineage capture component used in the described AWS Glue example.
• Lineage API: Interface through which Spline lineage information can be consumed.
• Amazon Neptune: Graph database used in the example to store lineage data.
• Schema evolution: Changing fields or columns over time without breaking existing systems.
• AWS Glue Schema Registry: Supports schema discovery, validation, registration, compatibility, and version management.

Flashcards

Q: An analytics team needs course enrollments with reusable course, student, and payment details. What modeling structure should it consider?

A: A star schema: place enrollment events in a central fact table and descriptive course, student, and payment information in dimension tables connected by keys.

Q: What is the decisive difference between a fact table and a dimension table in a star schema?

A: A fact table records business events or measurements, while a dimension table supplies descriptive context about entities associated with those events.

Q: When would an ERD be more appropriate than a lineage diagram?

A: Use an ERD to show entities, attributes, and key-based relationships in a data model. Use lineage to show how data moves and transforms through a pipeline.

Q: A reporting value is incorrect, and the team needs to identify which upstream transformation introduced the error. What capability is most relevant?

A: Data lineage, because it records the data’s path and transformations from source to destination.

Q: What does data lineage document across a data lifecycle?

A: It documents the source, movement, transformations, and final destination or consumption of data.

Q: In the described AWS Glue example, which component captures lineage from the processing activity?

A: Spline Agent, a third-party Apache Spark-related component attached to AWS Glue. Its captured information can be consumed through the Spline Lineage API.

Q: Why is Amazon Neptune a sensible destination for lineage information in the example?

A: Lineage naturally forms a graph of related sources, transformations, and destinations, and Neptune is a graph database suited to storing and querying such relationships.

Q: A business adds a field to incoming records but must continue processing older records. What concept addresses this requirement?

A: Schema evolution, which allows schemas to change while maintaining compatibility with older data and existing processes.

Q: What types of schema changes can schema evolution accommodate?

A: It can accommodate adding, removing, or modifying columns and fields, provided existing systems and consumers are not disrupted.

Q: When should AWS Glue Schema Registry be considered?

A: Consider it when schema versions, validation, discovery, registration, and compatibility between older and newer data need to be managed.

Q: How do data lineage and schema evolution differ?

A: Lineage tracks where data came from and how it was transformed; schema evolution manages how the structure of that data changes over time.

Q: A question describes Spline Agent attached to Glue and asks how another system can retrieve the captured lineage. What should you identify?

A: The Spline Lineage API, which exposes the captured lineage for consumption by Spline tools or downstream processing.

Q: Why can schema flexibility be useful in a data lake?

A: A data lake can accommodate changing data structures without requiring a disruptive migration every time a new business requirement introduces or changes fields.

Q: A new engineer needs to understand how a dataset moved through several pipeline stages. Which documentation would provide the most direct answer?

A: Data lineage, because it provides a clear record of the dataset’s movement, transformations, and consumption.

Practice Questions

Question 1

An e-learning company wants to analyze enrollments by course, student, and payment attributes. The enrollment record should remain compact while descriptive details are reusable. Which design best fits this requirement?

A. Store every course, student, and payment attribute directly in each enrollment row

B. Use a central enrollment fact table linked to course, student, and payment dimensions

C. Store only descriptive dimensions and reconstruct enrollments from them

D. Use a lineage graph as the analytical data model

Correct answer: B

Explanation: A star schema uses a central fact table for events such as enrollments and dimension tables for descriptive context, connected through keys.

Question 2

A pipeline reads raw data from Amazon S3 through AWS Glue, transforms it, and produces an output with an incorrect value. The engineer needs to trace the value through the processing steps. What should the engineer use?

A. An ERD only

B. AWS Glue Schema Registry only

C. Data lineage captured from the pipeline

D. A payment dimension table

Correct answer: C

Explanation: Lineage records the source and transformations applied to data, making it useful for tracing errors through a pipeline.

Question 3

A team uses Apache Spark processing in AWS Glue and wants to capture the processing history for visualization through a lineage interface. Which component matches the example in the lesson?

A. Spline Agent with its Lineage API

B. Amazon Neptune as the capture agent

C. An ERD generated from the fact table

D. AWS Glue Schema Registry as the lineage collector

Correct answer: A

Explanation: The described AWS pattern attaches Spline Agent to Glue processing, then exposes captured lineage through the Spline Lineage API.

Question 4

A data producer must introduce new fields while older records and existing consumers continue to work. Which capability addresses the primary concern?

A. Schema evolution managed with compatibility controls

B. Data lineage stored in an ERD

C. A payment dimension joined to the fact table

D. Replacing all historical records with the new schema

Correct answer: A

Explanation: Schema evolution permits structural changes over time while preserving compatibility with older data and processes. The AWS Glue Schema Registry supports schema version and compatibility management.

Question 5

An engineering team wants a store for lineage relationships among source datasets, transformations, and downstream outputs. Which option is most consistent with the lesson’s example?

A. Amazon Neptune

B. A course dimension table

C. An enrollment fact table

D. A schema field without version information

Correct answer: A

Explanation: Lineage has a graph-like structure, and the example uses Amazon Neptune, a graph database, to store and query that information.

WordPress Metadata

Suggested Slug:
aws-data-modeling-lineage-schema-evolution-review

Meta Description:
Review star schemas, data lineage, Spline Agent, Amazon Neptune, and AWS Glue Schema Registry for AWS Certified Data Engineer Associate preparation.

Tags:
AWS Certified Data Engineer Associate, data modeling, star schema, fact tables, dimension tables, entity relationship diagrams, data lineage, schema evolution, AWS Glue, Spline Agent, Amazon Neptune, Glue Schema Registry