Get Databricks Databricks-Certified-Data-Engineer-Associate Exam Dumps

Answer : E

Option E is the correct answer because it is the only mandatory requirement when creating a new Delta Live Tables pipeline. A pipeline is a data processing workflow that contains materialized views and streaming tables declared in Python or SQL source files. Delta Live Tables infers the dependencies between these tables and ensures updates occur in the correct order. To create a pipeline, you need to specify at least one notebook library to be executed, which contains the Delta Live Tables syntax. You can also specify multiple libraries of different languages within your pipeline. The other options are optional or not applicable for creating a pipeline. Option A is not required, but you can optionally provide a key-value pair configuration to customize the pipeline settings, such as the storage location, the target schema, the notifications, and the pipeline mode. Option B is not applicable, as the DBU/hour cost is determined by the cluster configuration, not the pipeline creation. Option C is not required, but you can optionally specify a storage location for the output data from the pipeline. If you leave it empty, the system uses a default location. Option D is not required, but you can optionally specify a location of a target database for the written data, either in the Hive metastore or the Unity Catalog.

Answer : C

To return the number of null values in the member_id column, the best option is to use the count_if function, which counts the number of rows that satisfy a given condition. In this case, the condition is that the member_id column is null. The other options are either incorrect or not supported by Spark SQL. Option A will return the number of non-null values in the member_id column. Option B will not work because there is no count_null function in Spark SQL. Option D will not work because there is no null function in Spark SQL. Option E will not work because there is no count_null function in Spark SQL.Reference:

Built-in Functions - Spark SQL, Built-in Functions

count_if - Spark SQL, Built-in Functions

Answer : B

dbfs:/user/hive/warehouse Thereby showing 'dbfs:/user/hive/warehouse/customer360.db

The location of the customer360 database depends on the value of thespark.sql.warehouse.dirconfiguration property, which specifies the default location for managed databases and tables. If the property is not set, the default value isdbfs:/user/hive/warehouse. Therefore, the customer360 database will be located indbfs:/user/hive/warehouse/customer360.db. However, if the property is set to a different value, such asdbfs:/user/hive/database, then the customer360 database will be located indbfs:/user/hive/database/customer360.db. Thus, more information is needed to determine the correct response.

Option A is not correct, asdbfs:/user/hive/database/customer360is not the default location for managed databases and tables, unless thespark.sql.warehouse.dirproperty is explicitly set todbfs:/user/hive/database.

Option B is not correct, asdbfs:/user/hive/warehouseis the default location for the root directory of managed databases and tables, not for a specific database. The database name should be appended with.dbto the directory path, such asdbfs:/user/hive/warehouse/customer360.db.

Option C is not correct, asdbfs:/user/hive/customer360is not a valid location for a managed database, as it does not follow the directory structure specified by thespark.sql.warehouse.dirproperty.

Databases and Tables

[Databricks Data Engineer Professional Exam Guide]

Answer : E

A data lakehouse is a new paradigm that can be used to simplify and unify siloed data architectures that are specialized for specific use cases. A data lakehouse combines the best of both data lakes and data warehouses, providing a single platform that supports diverse data types, open standards, low-cost storage, high-performance queries, ACID transactions, schema enforcement, and governance. A data lakehouse enables data engineers to build reliable and scalable data pipelines that can serve various downstream applications and users, such as data science, machine learning, analytics, and reporting. A data lakehouse leverages the power of Delta Lake, a storage layer that brings reliability and performance to data lakes.Reference:What is a data lakehouse?,Delta Lake,Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics

Answer : A

The scenario in which a data engineer will want to use a single-node cluster is when they are working interactively with a small amount of data.A single-node cluster is a cluster consisting of an Apache Spark driver and no Spark workers1.A single-node cluster supports Spark jobs and all Spark data sources, including Delta Lake1.A single-node cluster is helpful for single-node machine learning workloads that use Spark to load and save data, and for lightweight exploratory data analysis1.A single-node cluster can run Spark locally, spawn one executor thread per logical core in the cluster, and save all log output in the driver log1.A single-node cluster can be created by selecting the Single Node button when configuring a cluster1.

The other options are not suitable for using a single-node cluster.When running automated reports to be refreshed as quickly as possible, a data engineer will want to use a multi-node cluster that can scale up and down automatically based on the workload demand2.When working with SQL within Databricks SQL, a data engineer will want to use a SQL Endpoint that can execute SQL queries on a serverless pool or an existing cluster3.When concerned about the ability to automatically scale with larger data, a data engineer will want to use a multi-node cluster that can leverage the Databricks Lakehouse Platform and the Delta Engine to handle large-scale data processing efficiently and reliably4. When manually running reports with a large amount of data, a data engineer will want to use a multi-node cluster that can distribute the computation across multiple workers and leverage the Spark UI to monitor the performance and troubleshoot the issues.

1:Single Node clusters | Databricks on AWS

2:Autoscaling | Databricks on AWS

3:SQL Endpoints | Databricks on AWS

4:Databricks Lakehouse Platform | Databricks on AWS

: [Spark UI | Databricks on AWS]