[2023年12月] 今すぐダウンロード！リアルDatabricks Databricks-Certified-Professional-Data-Engineer試験問題集テストエンジン試験問題
最新Databricks-Certified-Professional-Data-Engineerテスト問題集を試そう！最新Databricks試験合格させます
Databricks認定のプロフェッショナルData-Engineer試験は、データパイプラインを設計、実装、および管理する候補者の能力を評価し、DataBricksプラットフォームで高度な分析と機械学習技術を活用する包括的な評価です。この試験は、複数の選択の質問で構成されており、候補者がDataBricksプラットフォーム上でデータソリューションを構築する能力を示す実践プロジェクトを完了することを要求します。
質問 # 23
Incorporating unit tests into a PySpark application requires upfront attention to the design of your jobs, or a potentially significant refactoring of existing code.
Which statement describes a main benefit that offset this additional effort?
- A. Ensures that all steps interact correctly to achieve the desired end result
- B. Troubleshooting is easier since all steps are isolated and tested individually
- C. Improves the quality of your data
- D. Yields faster deployment and execution times
- E. Validates a complete use case of your application
正解：B
質問 # 24
Which of the following developer operations in CI/CD flow can be implemented in Databricks Re-pos?
- A. Delete branch
- B. Approve the pull request
- C. Commit and push code
- D. Trigger Databricks CICD pipeline
- E. Create a pull request
正解：C
解説：
Explanation
The answer is Commit and push code.
See the below diagram to understand the role Databricks Repos and Git provider plays when building a CI/CD workflow.
All the steps highlighted in yellow can be done Databricks Repo, all the steps highlighted in Gray are done in a git provider like Github or Azure Devops.
Exam focus: Please study the below image carefully to understand all of the steps in the CI/CD flow to understand the tasks that are implemented in Databricks Repo vs Git Provider, exam may ask a different type of questions based on this flow.
Diagram Description automatically generated
質問 # 25
You were asked to create or overwrite an existing delta table to store the below transaction data.
- A. 1.CREATE OR REPLACE DELTA TABLE transactions (
2.transactionId int,
3.transactionDate timestamp,
4.unitsSold int) - B. 1.CREATE OR REPLACE TABLE IF EXISTS transactions (
2.transactionId int,
3.transactionDate timestamp,
4.unitsSold int)
5.FORMAT DELTA - C. 1.CREATE OR REPLACE TABLE transactions (
2.transactionId int,
3.transactionDate timestamp,
4.unitsSold int) - D. 1.CREATE IF EXSITS REPLACE TABLE transactions (
2.transactionId int,
3.transactionDate timestamp,
4.unitsSold int)
正解：C
解説：
Explanation
The answer is
1.CREATE OR REPLACE TABLE transactions (
2.transactionId int,
3.transactionDate timestamp,
4.unitsSold int)
When creating a table in Databricks by default the table is stored in DELTA format.
質問 # 26
A table is registered with the following code:
Bothusersandordersare Delta Lake tables. Which statement describes the results of queryingrecent_orders?
- A. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query began.
- B. The versions of each source table will be stored in the table transaction log; query results will be saved to DBFS with each query.
- C. Results will be computed and cached when the table is defined; these cached results will incrementally update as new records are inserted into source tables.
- D. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query finishes.
- E. All logic will execute when the table is definedand store the result of joiningtables to the DBFS; this stored data will be returned when the table is queried.
正解：A
解説：
Explanation
This is the correct answer because Delta Lake supports time travel, which allows users to query data as of a specific version or timestamp. The code uses the VERSION AS OF syntax to specify the version of each source table to be used in the join. The result of querying recent_orders will be the same as joining those versions of the source tables at query time. The query will use snapshot isolation, which means it will use a consistent snapshot of the table at the time the query began, regardless of any concurrent updates or deletes.
Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Query an older snapshot of a table (time travel)" section.
質問 # 27
A data architect has designed a system in which two Structured Streaming jobs will concurrently write to a single bronze Delta table. Each job is subscribing to a different topic from an Apache Kafka source, but they will write data with the same schema. To keep the directory structure simple, a data engineer has decided to nest a checkpoint directory to be shared by both streams.
The proposed directory structure is displayed below:
Which statement describes whether this checkpoint directory structure is valid for the given scenario and why?
- A. Yes; Delta Lake supports infinite concurrent writers.
- B. No; each of the streams needs to have its own checkpoint directory.
- C. No; Delta Lake manages streaming checkpoints in the transaction log.
- D. No; only one stream can write to a Delta Lake table.
- E. Yes; both of the streams can share a single checkpoint directory.
正解：B
解説：
Explanation
This is the correct answer because checkpointing is a critical feature of Structured Streaming that provides fault tolerance and recovery in case of failures. Checkpointing stores the current state and progress of a streaming query in a reliable storage system, such as DBFS or S3. Each streaming query must have its own checkpoint directory that is unique and exclusive to that query. If two streaming queries share the same checkpoint directory, they will interfere with each other and cause unexpected errors or data loss. Verified References: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "Checkpointing" section.
質問 # 28
Which of the following statements can be used to test the functionality of code to test number of rows in the table equal to 10 in python?
row_count = spark.sql("select count(*) from table").collect()[0][0]
- A. assert if (row_count = 10, "Row count did not match")
- B. assert (row_count = 10, "Row count did not match")
- C. assert if row_count == 10, "Row count did not match"
- D. assert row_count = 10, "Row count did not match"
- E. assert row_count == 10, "Row count did not match"
正解：E
解説：
Explanation
The answer is assert row_count == 10, "Row count did not match"
Review below documentation
質問 # 29
A data engineer has a Job with multiple tasks that runs nightly. One of the tasks unexpectedly fails during 10
percent of the runs.
Which of the following actions can the data engineer perform to ensure the Job completes each night while
minimizing compute costs?
- A. They can set up the Job to run multiple times ensuring that at least one will complete
- B. They can observe the task as it runs to try and determine why it is failing
- C. They can institute a retry policy for the task that periodically fails
- D. They can utilize a Jobs cluster for each of the tasks in the Job
- E. They can institute a retry policy for the entire Job
正解：C
質問 # 30
What is the output of below function when executed with input parameters 1, 3 :
1.def check_input(x,y):
2. if x < y:
3. x= x+1
4. if x>y:
5. x= x+1
6. if x <y:
7. x = x+1
8. return x
- A. 0
- B. 1
- C. 2
- D. 3
- E. 4
正解：E
質問 # 31
You are asked to setup two tasks in a databricks job, the first task runs a notebook to download the data from a remote system, and the second task is a DLT pipeline that can process this data, how do you plan to configure this in Jobs UI
- A. Jobs UI does not support DTL pipeline, setup the first task using jobs UI and setup the DLT to run in continuous mode.
- B. Jobs UI does not support DTL pipeline, setup the first task using jobs UI and setup the DLT to run in trigger mode.
- C. Single job can be used to setup both notebook and DLT pipeline, use two different tasks with linear dependency.
- D. Single job cannot have a notebook task and DLT Pipeline task, use two different jobs with linear dependency.
- E. Add first step in the DLT pipeline and run the DLT pipeline as triggered mode in JOBS UI
正解：C
解説：
Explanation
The answer is Single job can be used to set up both notebook and DLT pipeline, use two different tasks with linear dependency, Here is the JOB UI
1.Create a notebook task
2.Create DLT task
a.add notebook task as dependency
3.Final view
Create the notebook task
Graphical user interface, text, application, email Description automatically generated
DLT task
Graphical user interface, text, application, email Description automatically generated
Final view
Graphical user interface, text, application, PowerPoint Description automatically generated
Bottom of Form
Top of Form
質問 # 32
Which of the following scenarios is the best fit for the AUTO LOADER solution?
- A. Efficiently move data incrementally from one delta table to another delta table
- B. Incrementally process new data from relational databases like MySQL
- C. Incrementally process new streaming data from Apache Kafa into delta lake
- D. Efficiently process new data incrementally from cloud object storage
- E. Efficiently copy data from data lake location to another data lake location
正解：D
解説：
Explanation
The answer is, Efficiently process new data incrementally from cloud object storage.
Please note: AUTO LOADER only works on data/files located in cloud object storage like S3 or Azure Blob Storage it does not have the ability to read other data sources, although AU-TO LOADER is built on top of structured streaming it only supports files in the cloud object stor-age. If you want to use Apache Kafka then you can just use structured streaming.
Diagram Description automatically generated
Auto Loader and Cloud Storage Integration
Auto Loader supports a couple of ways to ingest data incrementally
1.Directory listing - List Directory and maintain the state in RocksDB, supports incremental file listing
2.File notification - Uses a trigger+queue to store the file notification which can be later used to retrieve the file, unlike Directory listing File notification can scale up to millions of files per day.
[OPTIONAL]
Auto Loader vs COPY INTO?
Auto Loader
Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage without any additional setup. Auto Loader provides a new Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files in that directory.
When to use Auto Loader instead of the COPY INTO?
*You want to load data from a file location that contains files in the order of millions or higher. Auto Loader can discover files more efficiently than the COPY INTO SQL command and can split file processing into multiple batches.
*You do not plan to load subsets of previously uploaded files. With Auto Loader, it can be more difficult to reprocess subsets of files. However, you can use the COPY INTO SQL command to reload subsets of files while an Auto Loader stream is simultaneously running.
Refer to more documentation here,
https://docs.microsoft.com/en-us/azure/databricks/ingestion/auto-loader
質問 # 33
Create a sales database using the DBFS location 'dbfs:/mnt/delta/databases/sales.db/'
- A. CREATE DELTA DATABASE sales LOCATION 'dbfs:/mnt/delta/databases/sales.db/'
- B. CREATE DATABASE sales USING LOCATION 'dbfs:/mnt/delta/databases/sales.db/'
- C. The sales database can only be created in Delta lake
- D. CREATE DATABASE sales LOCATION 'dbfs:/mnt/delta/databases/sales.db/'
- E. CREATE DATABASE sales FORMAT DELTA LOCATION 'dbfs:/mnt/delta/databases/sales.db/''
正解：C
解説：
Explanation
The answer is
CREATE DATABASE sales LOCATION 'dbfs:/mnt/delta/databases/sales.db/'
Note: with the introduction of the Unity catalog and three-layer namespace usage of SCHEMA and DATABASE is interchangeable
質問 # 34
How do you access or use tables in the unity catalog?
- A. catalog_name.table_name
- B. catalog_name.database_name.schema_name.table_name
- C. catalog_name.schema_name.table_name
- D. schema_name.catalog_name.table_name
- E. schema_name.table_name
正解：C
解説：
Explanation
The answer is catalog_name.schema_name.table_name
Graphical user interface, diagram Description automatically generated
Note: Database and Schema are analogous they are interchangeably used in the Unity catalog.
FYI, A catalog is registered under a metastore, by default every workspace has a default metastore called hive_metastore, with a unity catalog you have the ability to create meatstores and share that across multiple workspaces.
Diagram Description automatically generated
質問 # 35
Data engineering team has a job currently setup to run a task load data into a reporting table every day at 8: 00 AM takes about 20 mins, Operations teams are planning to use that data to run a second job, so they access latest complete set of data. What is the best to way to orchestrate this job setup?
- A. Use Auto Loader to run every 20 mins to read the initial table and set the trigger to once and create a second job
- B. Add Operation reporting task in the same job and set the Data Engineering task to de-pend on Operations reporting task
- C. Add Operation reporting task in the same job and set the operations reporting task to depend on Data Engineering task
- D. Setup a Delta live to table based on the first table, set the job to run in continuous mode
- E. Setup a second job to run at 8:20 AM in the same workspace
正解：C
解説：
Explanation
The answer is Add Operation reporting task in the same job and set the operations reporting task to depend on Data Engineering task.
Diagram Description automatically generated with medium confidence
質問 # 36
When using the complete mode to write stream data, how does it impact the target table?
- A. Target table is overwritten for each batch
- B. Stream must complete to write the data
- C. Target table cannot be updated while stream is pending
- D. Entire stream waits for complete data to write
- E. Delta commits transaction once the stream is stopped
正解：A
解説：
Explanation
The answer is Target table is overwritten for each batch
Complete mode - The whole Result Table will be outputted to the sink after every trigger. This is supported for aggregation queries
質問 # 37
Which of the following is true, when building a Databricks SQL dashboard?
- A. A dashboard can only connect to one schema/Database
- B. A dashboard can only have one refresh schedule
- C. More than one visualization can be developed using a single query result
- D. Only one visualization can be developed with one query result
- E. A dashboard can only use results from one query
正解：C
解説：
Explanation
the answer is, More than one visualization can be developed using a single query result.
In the query editor pane + Add visualization tab can be used for many visualizations for a single query result.
Graphical user interface, text, application Description automatically generated
質問 # 38
Question-26. There are 5000 different color balls, out of which 1200 are pink color. What is the maximum
likelihood estimate for the proportion of "pink" items in the test set of color balls?
- A. .48
- B. 2.4
- C. 4.8
- D. 24 0
- E. .24
正解：E
解説：
Explanation
Given no additional information, the MLE for the probability of an item in the test set is exactly its frequency
in the training set. The method of maximum likelihood corresponds to many well-known estimation methods
in statistics. For example, one may be interested in the heights of adult female penguins, but be unable to
measure the height of every single penguin in a population due to cost or time constraints. Assuming that the
heights are normally (Gaussian) distributed with some unknown mean and variance, the mean and variance
can be estimated with MLE while only knowing the heights of some sample of the overall population. MLE
would accomplish this by taking the mean and variance as parameters and finding particular parametric values
that make the observed results the most probable (given the model).
In general, for a fixed set of data and underlying statistical model the method of maximum likelihood selects
the set of values of the model parameters that maximizes the likelihood function. Intuitively, this maximizes
the "agreement" of the selected model with the observed data, and for discrete random variables it indeed
maximizes the probability of the observed data under the resulting distribution. Maximum-likelihood
estimation gives a unified approach to estimation, which is well-defined in the case of the normal distribution
and many other problems. However in some complicated problems, difficulties do occur: in such problems,
maximum-likelihood estimators are unsuitable or do not exist.
質問 # 39
Which of the following features of data lakehouse can help you meet the needs of both workloads?
- A. Data lakehouse can store unstructured data and support ACID transactions.
- B. Data lakehouse fully exists in the cloud.
- C. Data lakehouse combines compute and storage for simple governance.
- D. Data lakehouse requires very little data modeling.
- E. Data lakehouse provides autoscaling for compute clusters.
正解：A
解説：
Explanation
The answer is A data lakehouse stores unstructured data and is ACID-compliant,
質問 # 40
Which statement characterizes the general programming model used by Spark Structured Streaming?
- A. Structured Streaming relies on a distributed network of nodes that hold incremental state values for cached stages.
- B. Structured Streaming leverages the parallel processing of GPUs to achieve highly parallel data throughput.
- C. Structured Streaming models new data arriving in a data stream as new rows appended to an unbounded table.
- D. Structured Streaming is implemented as a messaging bus and is derived from Apache Kafka.
- E. Structured Streaming uses specialized hardware and I/O streams to achieve sub-second latency for data transfer.
正解：C
解説：
Explanation
This is the correct answer because it characterizes the general programming model used by Spark Structured Streaming, which is to treat a live data stream as a table that is being continuously appended. This leads to a new stream processing model that is very similar to a batch processing model, where users can express their streaming computation using the same Dataset/DataFrame API as they would use for static data. The Spark SQL engine will take care of running the streaming query incrementally and continuously and updating the final result as streaming data continues to arrive. Verified References: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "Overview" section.
質問 # 41
A denote the event 'student is female' and let B denote the event 'student is French'. In a class of 100 students
suppose 60 are French, and suppose that 10 of the French students are females. Find the probability that if I
pick a French student, it will be a girl, that is, find P(A|B).
- A. 2/3
- B. 2/6
- C. 1/3
- D. 1/6
正解：D
解説：
Explanation
Since 10 out of 100 students are both French and female, then
P(AandB)=10100
Also. 60 out of the 100 students are French, so
P(B)=60100
So the required probability is:
P(A|B)=P(AandB)P(B)=10/10060/100=16
質問 # 42
The data engineering team maintains a table of aggregate statistics through batch nightly updates. This includes total sales for the previous day alongside totals and averages for a variety of time periods including the 7 previous days, year-to-date, and quarter-to-date. This table is namedstore_saies_summaryand the schema is as follows:
The tabledaily_store_salescontains all the information needed to updatestore_sales_summary. The schema for this table is:
store_id INT, sales_date DATE, total_sales FLOAT
Ifdaily_store_salesis implemented as a Type 1 table and the column might be adjusted after manual data auditing, which approach is the safest to generate accurate reports in thestore_sales_summarytable?
- A. Implement the appropriate aggregate logic as a batch read against the daily_store_sales table and use upsert logic to update results in the store_sales_summary table.
- B. Implement the appropriate aggregate logic as a batch read against the daily_store_sales table and append new rows nightly to the store_sales_summary table.
- C. Use Structured Streaming to subscribe to the change data feed for daily_store_sales and apply changes to the aggregates in the store_sales_summary table with each update.
- D. Implement the appropriate aggregate logic as a Structured Streaming read against the daily_store_sales table and use upsert logic to update results in the store_sales_summary table.
- E. Implement the appropriate aggregate logic as a batch read against the daily_store_sales table and overwrite the store_sales_summary table with each Update.
正解：C
解説：
Explanation
The daily_store_sales table contains all the information needed to update store_sales_summary. The schema of the table is:
store_id INT, sales_date DATE, total_sales FLOAT
The daily_store_sales table is implemented as a Type 1 table, which means that old values are overwritten by new values and no history is maintained. The total_sales column might be adjusted after manual data auditing, which means that the data in the table may change over time.
The safest approach to generate accurate reports in the store_sales_summary table is to use Structured Streaming to subscribe to the change data feed for daily_store_sales and apply changes to the aggregates in the store_sales_summary table with each update. Structured Streaming is a scalable and fault-tolerant stream processing engine built on Spark SQL. Structured Streaming allows processing data streams as if they were tables or DataFrames, using familiar operations such as select, filter, groupBy, or join. Structured Streaming also supports output modes that specify how to write the results of a streaming query to a sink, such as append, update, or complete. Structured Streaming can handle both streaming and batch data sources in a unified manner.
The change data feed is a feature of Delta Lake that provides structured streaming sources that can subscribe to changes made to a Delta Lake table. The change data feed captures both data changes and schema changes as ordered events that can be processed by downstream applications or services. The change data feed can be configured with different options, such as starting from a specific version or timestamp, filtering by operation type or partition values, or excluding no-op changes.
By using Structured Streaming to subscribe to the change data feed for daily_store_sales, one can capture and process any changes made to the total_sales column due to manual data auditing. By applying these changes to the aggregates in the store_sales_summary table with each update, one can ensure that the reports are always consistent and accurate with the latest data. Verified References: [Databricks Certified Data Engineer Professional], under "Spark Core" section; Databricks Documentation, under "Structured Streaming" section; Databricks Documentation, under "Delta Change Data Feed" section.
質問 # 43
Which of the following commands results in the successful creation of a view on top of the delta stream(stream on delta table)?
- A. Spark.read.format("delta").table("sales").trigger("stream").createOrReplaceTempView("streaming_vw")
- B. Spark.readStream.format("delta").table("sales").createOrReplaceTempView("streaming_vw")
- C. Spark.read.format("delta").table("sales").createOrReplaceTempView("streaming_vw")
- D. Spark.read.format("delta").stream("sales").createOrReplaceTempView("streaming_vw")
- E. Spark.read.format("delta").table("sales").mode("stream").createOrReplaceTempView("streaming_vw")
- F. You can not create a view on streaming data source.
正解：B
解説：
Explanation
The answer is
Spark.readStream.table("sales").createOrReplaceTempView("streaming_vw") When you load a Delta table as a stream source and use it in a streaming query, the query processes all of the data present in the table as well as any new data that arrives after the stream is started.
You can load both paths and tables as a stream, you also have the ability to ignore deletes and changes(updates, Merge, overwrites) on the delta table.
Here is more information,
https://docs.databricks.com/delta/delta-streaming.html#delta-table-as-a-source
質問 # 44
You are noticing job cluster is taking 6 to 8 mins to start which is delaying your job to finish on time, what steps you can take to reduce the amount of time cluster startup time
- A. Use SQL endpoints to reduce the startup time
- B. Reduce the size of the cluster, smaller the cluster size shorter it takes to start the clus-ter
- C. Setup a second job ahead of first job to start the cluster, so the cluster is ready with re-sources when the job starts
- D. Use cluster pools to reduce the startup time of the jobs
- E. Use All purpose cluster instead to reduce cluster start up time
正解：D
解説：
Explanation
The answer is, Use cluster pools to reduce the startup time of the jobs.
Cluster pools allow us to reserve VM's ahead of time, when a new job cluster is created VM are grabbed from the pool. Note: when the VM's are waiting to be used by the cluster only cost incurred is Azure. Databricks run time cost is only billed once VM is allocated to a cluster.
Here is a demo of how to setup and follow some best practices,
https://www.youtube.com/watch?v=FVtITxOabxg&ab_channel=DatabricksAcademy
質問 # 45
......
DCPDE試験の準備をするには、候補者は、データモデリング、データ統合、データ変換、データ品質などのデータエンジニアリングの概念を確実に理解する必要があります。また、Apache Spark、Apache Kafka、Apache Hadoopなどのビッグデータテクノロジーの操作経験もあります。
あなたを必ず合格させるDatabricks-Certified-Professional-Data-Engineer問題集PDF2023年最新のに更新された60問あります：https://jp.fast2test.com/Databricks-Certified-Professional-Data-Engineer-premium-file.html
検証済み！Databricks-Certified-Professional-Data-Engineer問題集と解答でDatabricks-Certified-Professional-Data-Engineerテストエンジン正確解答付き：https://drive.google.com/open?id=12CWU92nghYtJUIefz-X5MLwwWIodKH1J