Databricks Databricks-Certified-Professional-Data-Engineer Practice Test Questions Answers

The provided PySpark code performs the following operations:

Reads Data from silver_customer_sales Table:

The code starts by accessing the silver_customer_sales table using the spark.table method.

Groups Data by customer_id:

The .groupBy( " customer_id " ) function groups the data based on the customer_id column.

Aggregates Data:

The .agg() function computes several aggregate metrics for each customer_id:

F.min( " sale_date " ).alias( " first_transaction_date " ): Determines the earliest sale date for the customer.

F.max( " sale_date " ).alias( " last_transaction_date " ): Determines the latest sale date for the customer.

F.mean( " sale_total " ).alias( " average_sales " ): Calculates the average sale amount for the customer.

F.countDistinct( " order_id " ).alias( " total_orders " ): Counts the number of unique orders placed by the customer.

F.sum( " sale_total " ).alias( " lifetime_value " ): Calculates the total sales amount (lifetime value) for the customer.

Writes Data to gold_customer_lifetime_sales_summary Table:

The .write.mode( " overwrite " ).table( " gold_customer_lifetime_sales_summary " ) command writes the aggregated data to the gold_customer_lifetime_sales_summary table.

The mode( " overwrite " ) specifies that the existing data in the gold_customer_lifetime_sales_summary table will be completely replaced by the new aggregated data.

Conclusion:

When this code is executed, it reads all records from the silver_customer_sales table, performs the specified aggregations grouped by customer_id, and then overwrites the entire gold_customer_lifetime_sales_summary table with the aggregated results. Therefore, option D accurately describes this process: " The gold_customer_lifetime_sales_summary table will be overwritten by aggregated values calculated from all records in the silver_customer_sales table as a batch job. "

This is the correct answer because it describes a limitation of Databricks Secrets. Databricks Secrets is a module that provides tools to store sensitive credentials and avoid accidentally displaying them in plain text. Databricks Secrets allows creating secret scopes, which are collections of secrets that can be accessed by users or groups. Databricks Secrets also allows creating and managing secrets using the Databricks CLI or the Databricks REST API. However, a limitation of Databricks Secrets is that the Databricks REST API can be used to list secrets in plain text if the personal access token has proper credentials. Therefore, users should still be careful with which credentials are stored in Databricks Secrets and which users have access to using these secrets. Verified References: [Databricks Certified Data Engineer Professional], under “Databricks Workspace” section; Databricks Documentation, under “List secrets” section.

The Databricks CLI allows users to manage clusters using command-line commands. The correct command for creating a cluster follows a specific format.

Key Components in the Command:

Command Type: databricks compute create is the correct syntax for creating a new compute resource (cluster).

Runtime Version: ' 14.3.x-scala2.12 ' specifies the Databricks runtime to use.

Workers: --num-workers 5 sets the number of worker nodes to 5.

Node Type: --node-type-id i3.xlarge defines the hardware configuration.

Cluster Name: --cluster-name DataEngineer_cluster assigns a recognizable name to the cluster.

Evaluation of Options:

Option A (databricks clusters create ...)

Incorrect: databricks clusters create is not a valid command in the Databricks CLI v0.205.

The correct CLI command for cluster creation is databricks compute create.

Option B (databricks clusters add ...)

Incorrect: databricks clusters add is not a valid CLI command .

Option C (databricks compute add ...)

Incorrect: databricks compute add is not a valid CLI command .

Option D (databricks compute create ...)

Correct: databricks compute create is the correct command for creating a cluster.

Conclusion:

The correct command to create a cluster with five workers, an i3.xlarge node type, and Databricks runtime 14.3.x-scala2.12 is:

databricks compute create 14.3.x-scala2.12 --num-workers 5 --node-type-id i3.xlarge --cluster-name Data Engineer_cluster

Thus, the correct answer is D .

Databricks recommends applying data masking or anonymization before persisting PII to ensure compliance with privacy regulations such as GDPR and HIPAA. In a Lakeflow Declarative Pipeline , developers can define custom Python or SQL-based masking functions to standardize PII handling across both batch and streaming inputs.

This approach ensures that data entering the Delta Lake is already anonymized, guaranteeing consistent and auditable behavior. By applying masking during ingestion (in the Bronze layer), audit trails are preserved through pipeline event logs.

While Unity Catalog column masks (option C) can enforce dynamic masking at query time, they do not prevent PII storage. Thus, option D aligns with the best practice of securing PII before storage , while still supporting reproducibility and analytics usability.

https://learn.microsoft.com/en-us/azure/databricks/delta/vacuum