[2023年04月]更新のDatabricks Associate-Developer-Apache-Spark試験基本問題には解答が付きます
2023年最新の実際に出るDatabricks Associate-Developer-Apache-Spark試験問題集と解答
Databricks Associate-Developer-Apache-Spark 認定試験は、Apache Spark技術に関するスキルと専門知識を検証するための絶好の機会です。これはまた、組織が従業員がApache Spark技術を扱うために必要なスキルを持っているかどうかを確認するための素晴らしい方法でもあります。この認定試験は2年間有効であり、その後は候補者が更新する必要があります。全体的に、この認定試験は、ビッグデータと分析の分野でキャリアの展望を向上させたいプロフェッショナルにとって、貴重な資格となります。
Databricks Associate-Developer-Apache-Spark認定試験は、オープンソースの分散コンピューティングシステムの最新バージョンであるApache Spark 3.0に基づいています。この試験は、Sparkアーキテクチャ、Spark SQL、Spark Streaming、Spark MLlib、Spark GraphXなど、Apache Sparkに関連する幅広いトピックをカバーしています。この試験は、Databricksを使用してApache Sparkアプリケーションを開発および展開するための候補者の知識と能力をテストするために設計されています。
質問 # 70
Which of the following options describes the responsibility of the executors in Spark?
- A. The executors accept jobs from the driver, plan those jobs, and return results to the cluster manager.
- B. The executors accept tasks from the driver, execute those tasks, and return results to the driver.
- C. The executors accept jobs from the driver, analyze those jobs, and return results to the driver.
- D. The executors accept tasks from the driver, execute those tasks, and return results to the cluster manager.
- E. The executors accept tasks from the cluster manager, execute those tasks, and return results to the driver.
正解：B
解説：
Explanation
More info: Running Spark: an overview of Spark's runtime architecture - Manning (https://bit.ly/2RPmJn9)
質問 # 71
The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.
Code block:
transactionsDf.coalesce(14, ("storeId", "transactionDate"))
- A. Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.
- B. The parentheses around the column names need to be removed and .select() needs to be appended to the code block.
- C. Operator coalesce needs to be replaced by repartition.
- D. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.
- E. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
(Correct)
正解：E
解説：
Explanation
Correct code block:
transactionsDf.repartition(14, "storeId", "transactionDate").count()
Since we do not know how many partitions DataFrame transactionsDf has, we cannot safely use coalesce, since it would not make any change if the current number of partitions is smaller than 14.
So, we need to use repartition.
In the Spark documentation, the call structure for repartition is shown like this:
DataFrame.repartition(numPartitions, *cols). The * operator means that any argument after numPartitions will be interpreted as column. Therefore, the brackets need to be removed.
Finally, the question specifies that after the execution the DataFrame should be divided. So, indirectly this question is asking us to append an action to the code block. Since .select() is a transformation. the only possible choice here is .count().
More info: pyspark.sql.DataFrame.repartition - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 # 72
The code block shown below should show information about the data type that column storeId of DataFrame transactionsDf contains. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(__2__).__3__
- A. 1. select
2. "storeId"
3. printSchema() - B. 1. select
2. storeId
3. dtypes - C. 1. limit
2. "storeId"
3. printSchema() - D. 1. limit
2. 1
3. columns - E. 1. select
2. "storeId"
3. print_schema()
正解：D
解説：
Explanation
Correct code block:
transactionsDf.select("storeId").printSchema()
The difficulty of this question is that it is hard to solve with the stepwise first-to-last-gap approach that has worked well for similar questions, since the answer options are so different from one another. Instead, you might want to eliminate answers by looking for patterns of frequently wrong answers.
A first pattern that you may recognize by now is that column names are not expressed in quotes. For this reason, the answer that includes storeId should be eliminated.
By now, you may have understood that the DataFrame.limit() is useful for returning a specified amount of rows. It has nothing to do with specific columns. For this reason, the answer that resolves to limit("storeId") can be eliminated.
Given that we are interested in information about the data type, you should question whether the answer that resolves to limit(1).columns provides you with this information. While DataFrame.columns is a valid call, it will only report back column names, but not column types. So, you can eliminate this option.
The two remaining options either use the printSchema() or print_schema() command. You may remember that DataFrame.printSchema() is the only valid command of the two. The select("storeId") part just returns the storeId column of transactionsDf - this works here, since we are only interested in that column's type anyways.
More info: pyspark.sql.DataFrame.printSchema - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
質問 # 73
The code block displayed below contains an error. The code block should return a DataFrame where all entries in column supplier contain the letter combination et in this order. Find the error.
Code block:
itemsDf.filter(Column('supplier').isin('et'))
- A. The Column operator should be replaced by the col operator and instead of isin, contains should be used.
- B. The expression only returns a single column and filter should be replaced by select.
- C. Instead of isin, it should be checked whether column supplier contains the letters et, so isin should be replaced with contains. In addition, the column should be accessed using col['supplier'].
- D. The expression inside the filter parenthesis is malformed and should be replaced by isin('et', 'supplier').
正解：D
解説：
Explanation
Correct code block:
itemsDf.filter(col('supplier').contains('et'))
A mixup can easily happen here between isin and contains. Since we want to check whether a column
"contains" the values et, this is the operator we should use here. Note that both methods are methods of Spark's Column object. See below for documentation links.
A specific Column object can be accessed through the col() method and not the Column() method or through col[], which is an essential thing to know here. In PySpark, Column references a generic column object. To use it for queries, you need to link the generic column object to a specific DataFrame. This can be achieved, for example, through the col() method.
More info:
- isin documentation: pyspark.sql.Column.isin - PySpark 3.1.1 documentation
- contains documentation: pyspark.sql.Column.contains - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 # 74
Which of the following code blocks returns a DataFrame with approximately 1,000 rows from the 10,000-row DataFrame itemsDf, without any duplicates, returning the same rows even if the code block is run twice?
- A. itemsDf.sample(fraction=1000, seed=98263)
- B. itemsDf.sampleBy("row", fractions={0: 0.1}, seed=82371)
- C. itemsDf.sample(fraction=0.1)
- D. itemsDf.sample(withReplacement=True, fraction=0.1, seed=23536)
- E. itemsDf.sample(fraction=0.1, seed=87238)
正解：E
解説：
Explanation
itemsDf.sample(fraction=0.1, seed=87238)
Correct. If itemsDf has 10,000 rows, this code block returns about 1,000, since DataFrame.sample() is never guaranteed to return an exact amount of rows. To ensure you are not returning duplicates, you should leave the withReplacement parameter at False, which is the default. Since the question specifies that the same rows should be returned even if the code block is run twice, you need to specify a seed. The number passed in the seed does not matter as long as it is an integer.
itemsDf.sample(withReplacement=True, fraction=0.1, seed=23536)
Incorrect. While this code block fulfills almost all requirements, it may return duplicates. This is because withReplacement is set to True.
Here is how to understand what replacement means: Imagine you have a bucket of 10,000 numbered balls and you need to take 1,000 balls at random from the bucket (similar to the problem in the question). Now, if you would take those balls with replacement, you would take a ball, note its number, and put it back into the bucket, meaning the next time you take a ball from the bucket there would be a chance you could take the exact same ball again. If you took the balls without replacement, you would leave the ball outside the bucket and not put it back in as you take the next 999 balls.
itemsDf.sample(fraction=1000, seed=98263)
Wrong. The fraction parameter needs to have a value between 0 and 1. In this case, it should be 0.1, since
1,000/10,000 = 0.1.
itemsDf.sampleBy("row", fractions={0: 0.1}, seed=82371)
No, DataFrame.sampleBy() is meant for stratified sampling. This means that based on the values in a column in a DataFrame, you can draw a certain fraction of rows containing those values from the DataFrame (more details linked below). In the scenario at hand, sampleBy is not the right operator to use because you do not have any information about any column that the sampling should depend on.
itemsDf.sample(fraction=0.1)
Incorrect. This code block checks all the boxes except that it does not ensure that when you run it a second time, the exact same rows will be returned. In order to achieve this, you would have to specify a seed.
More info:
- pyspark.sql.DataFrame.sample - PySpark 3.1.2 documentation
- pyspark.sql.DataFrame.sampleBy - PySpark 3.1.2 documentation
- Types of Samplings in PySpark 3. The explanations of the sampling... | by Pinar Ersoy | Towards Data Science
質問 # 75
Which of the following statements about executors is correct, assuming that one can consider each of the JVMs working as executors as a pool of task execution slots?
- A. Tasks run in parallel via slots.
- B. Slot is another name for executor.
- C. There must be more slots than tasks.
- D. There must be less executors than tasks.
- E. An executor runs on a single core.
正解：A
解説：
Explanation
Tasks run in parallel via slots.
Correct. Given the assumption, an executor then has one or more "slots", defined by the equation spark.executor.cores / spark.task.cpus. With the executor's resources divided into slots, each task takes up a slot and multiple tasks can be executed in parallel.
Slot is another name for executor.
No, a slot is part of an executor.
An executor runs on a single core.
No, an executor can occupy multiple cores. This is set by the spark.executor.cores option.
There must be more slots than tasks.
No. Slots just process tasks. One could imagine a scenario where there was just a single slot for multiple tasks, processing one task at a time. Granted - this is the opposite of what Spark should be used for, which is distributed data processing over multiple cores and machines, performing many tasks in parallel.
There must be less executors than tasks.
No, there is no such requirement.
More info: Spark Architecture | Distributed Systems Architecture (https://bit.ly/3x4MZZt)
質問 # 76
Which of the following code blocks creates a new 6-column DataFrame by appending the rows of the
6-column DataFrame yesterdayTransactionsDf to the rows of the 6-column DataFrame todayTransactionsDf, ignoring that both DataFrames have different column names?
- A. todayTransactionsDf.unionByName(yesterdayTransactionsDf)
- B. todayTransactionsDf.concat(yesterdayTransactionsDf)
- C. todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True)
- D. union(todayTransactionsDf, yesterdayTransactionsDf)
- E. todayTransactionsDf.union(yesterdayTransactionsDf)
正解：E
解説：
Explanation
todayTransactionsDf.union(yesterdayTransactionsDf)
Correct. The union command appends rows of yesterdayTransactionsDf to the rows of todayTransactionsDf, ignoring that both DataFrames have different column names. The resulting DataFrame will have the column names of DataFrame todayTransactionsDf.
todayTransactionsDf.unionByName(yesterdayTransactionsDf)
No. unionByName specifically tries to match columns in the two DataFrames by name and only appends values in columns with identical names across the two DataFrames. In the form presented above, the command is a great fit for joining DataFrames that have exactly the same columns, but in a different order. In this case though, the command will fail because the two DataFrames have different columns.
todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True) No. The unionByName command is described in the previous explanation. However, with the allowMissingColumns argument set to True, it is no longer an issue that the two DataFrames have different column names. Any columns that do not have a match in the other DataFrame will be filled with null where there is no value. In the case at hand, the resulting DataFrame will have 7 or more columns though, so it this command is not the right answer.
union(todayTransactionsDf, yesterdayTransactionsDf)
No, there is no union method in pyspark.sql.functions.
todayTransactionsDf.concat(yesterdayTransactionsDf)
Wrong, the DataFrame class does not have a concat method.
More info: pyspark.sql.DataFrame.union - PySpark 3.1.2 documentation,
pyspark.sql.DataFrame.unionByName - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 # 77
Which of the following DataFrame methods is classified as a transformation?
- A. DataFrame.foreach()
- B. DataFrame.select()
- C. DataFrame.show()
- D. DataFrame.count()
- E. DataFrame.first()
正解：B
解説：
Explanation
DataFrame.select()
Correct, DataFrame.select() is a transformation. When the command is executed, it is evaluated lazily and returns an RDD when it is triggered by an action.
DataFrame.foreach()
Incorrect, DataFrame.foreach() is not a transformation, but an action. The intention of foreach() is to apply code to each element of a DataFrame to update accumulator variables or write the elements to external storage. The process does not return an RDD - it is an action!
DataFrame.first()
Wrong. As an action, DataFrame.first() executed immediately and returns the first row of a DataFrame.
DataFrame.count()
Incorrect. DataFrame.count() is an action and returns the number of rows in a DataFrame.
DataFrame.show()
No, DataFrame.show() is an action and displays the DataFrame upon execution of the command.
質問 # 78
Which of the following statements about lazy evaluation is incorrect?
- A. Accumulators do not change the lazy evaluation model of Spark.
- B. Execution is triggered by transformations.
- C. Lineages allow Spark to coalesce transformations into stages
- D. Predicate pushdown is a feature resulting from lazy evaluation.
- E. Spark will fail a job only during execution, but not during definition.
正解：B
解説：
Explanation
Execution is triggered by transformations.
Correct. Execution is triggered by actions only, not by transformations.
Lineages allow Spark to coalesce transformations into stages.
Incorrect. In Spark, lineage means a recording of transformations. This lineage enables lazy evaluation in Spark.
Predicate pushdown is a feature resulting from lazy evaluation.
Wrong. Predicate pushdown means that, for example, Spark will execute filters as early in the process as possible so that it deals with the least possible amount of data in subsequent transformations, resulting in a performance improvements.
Accumulators do not change the lazy evaluation model of Spark.
Incorrect. In Spark, accumulators are only updated when the query that refers to the is actually executed. In other words, they are not updated if the query is not (yet) executed due to lazy evaluation.
Spark will fail a job only during execution, but not during definition.
Wrong. During definition, due to lazy evaluation, the job is not executed and thus certain errors, for example reading from a non-existing file, cannot be caught. To be caught, the job needs to be executed, for example through an action.
質問 # 79
The code block shown below should write DataFrame transactionsDf as a parquet file to path storeDir, using brotli compression and replacing any previously existing file. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__.format("parquet").__2__(__3__).option(__4__, "brotli").__5__(storeDir)
- A. 1. write
2. mode
3. "overwrite"
4. "compression"
5. save
(Correct) - B. 1. save
2. mode
3. "ignore"
4. "compression"
5. path - C. 1. write
2. mode
3. "overwrite"
4. compression
5. parquet - D. 1. save
2. mode
3. "replace"
4. "compression"
5. path - E. 1. store
2. with
3. "replacement"
4. "compression"
5. path
正解：D
解説：
Explanation
Correct code block:
transactionsDf.write.format("parquet").mode("overwrite").option("compression", "snappy").save(storeDir) Solving this question requires you to know how to access the DataFrameWriter (link below) from the DataFrame API - through DataFrame.write.
Another nuance here is about knowing the different modes available for writing parquet files that determine Spark's behavior when dealing with existing files. These, together with the compression options are explained in the DataFrameWriter.parquet documentation linked below.
Finally, bracket __5__ poses a certain challenge. You need to know which command you can use to pass down the file path to the DataFrameWriter. Both save and parquet are valid options here.
More info:
- DataFrame.write: pyspark.sql.DataFrame.write - PySpark 3.1.1 documentation
- DataFrameWriter.parquet: pyspark.sql.DataFrameWriter.parquet - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 # 80
Which of the following code blocks reorders the values inside the arrays in column attributes of DataFrame itemsDf from last to first one in the alphabet?
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+
- A. itemsDf.withColumn('attributes', sort(col('attributes'), asc=False))
- B. itemsDf.withColumn('attributes', sort_array(desc('attributes')))
- C. itemsDf.select(sort_array("attributes"))
- D. itemsDf.withColumn("attributes", sort_array("attributes", asc=False))
- E. itemsDf.withColumn('attributes', sort_array(col('attributes').desc()))
正解：D
解説：
Explanation
Output of correct code block:
+------+-----------------------------+-------------------+
|itemId|attributes |supplier |
+------+-----------------------------+-------------------+
|1 |[winter, cozy, blue] |Sports Company Inc.|
|2 |[summer, red, fresh, cooling]|YetiX |
|3 |[travel, summer, green] |Sports Company Inc.|
+------+-----------------------------+-------------------+
It can be confusing to differentiate between the different sorting functions in PySpark. In this case, a particularity about sort_array has to be considered: The sort direction is given by the second argument, not by the desc method. Luckily, this is documented in the documentation (link below). Also, for solving this question you need to understand the difference between sort and sort_array. With sort, you cannot sort values in arrays. Also, sort is a method of DataFrame, while sort_array is a method of pyspark.sql.functions.
More info: pyspark.sql.functions.sort_array - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2
質問 # 81
Which of the following code blocks produces the following output, given DataFrame transactionsDf?
Output:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- productId: integer (nullable = true)
7. |-- f: integer (nullable = true)
DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId| f|
3.+-------------+---------+-----+-------+---------+----+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.+-------------+---------+-----+-------+---------+----+
- A. transactionsDf.rdd.formatSchema()
- B. transactionsDf.schema.print()
- C. print(transactionsDf.schema)
- D. transactionsDf.printSchema()
- E. transactionsDf.rdd.printSchema()
正解：D
解説：
Explanation
The output is the typical output of a DataFrame.printSchema() call. The DataFrame's RDD representation does not have a printSchema or formatSchema method (find available methods in the RDD documentation linked below). The output of print(transactionsDf.schema) is this:
StructType(List(StructField(transactionId,IntegerType,true),StructField(predError,IntegerType,true),StructField (value,IntegerType,true),StructField(storeId,IntegerType,true),StructField(productId,IntegerType,true),StructFiel It includes the same information as the nicely formatted original output, but is not nicely formatted itself. Lastly, the DataFrame's schema attribute does not have a print() method.
More info:
- pyspark.RDD: pyspark.RDD - PySpark 3.1.2 documentation
- DataFrame.printSchema(): pyspark.sql.DataFrame.printSchema - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2
質問 # 82
Which of the following code blocks returns a new DataFrame with the same columns as DataFrame transactionsDf, except for columns predError and value which should be removed?
- A. transactionsDf.drop(["predError", "value"])
- B. transactionsDf.drop("predError & value")
- C. transactionsDf.drop("predError", "value")
- D. transactionsDf.drop(col("predError"), col("value"))
- E. transactionsDf.drop(predError, value)
正解：C
解説：
Explanation
More info: pyspark.sql.DataFrame.drop - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
質問 # 83
Which of the following code blocks reads in parquet file /FileStore/imports.parquet as a DataFrame?
- A. spark.read().parquet("/FileStore/imports.parquet")
- B. spark.mode("parquet").read("/FileStore/imports.parquet")
- C. spark.read().format('parquet').open("/FileStore/imports.parquet")
- D. spark.read.parquet("/FileStore/imports.parquet")
- E. spark.read.path("/FileStore/imports.parquet", source="parquet")
正解：D
解説：
Explanation
Static notebook | Dynamic notebook: See test 1
(https://flrs.github.io/spark_practice_tests_code/#1/23.html ,
https://bit.ly/sparkpracticeexams_import_instructions)
質問 # 84
Which of the following code blocks prints out in how many rows the expression Inc. appears in the string-type column supplier of DataFrame itemsDf?
- A. print(itemsDf.foreach(lambda x: 'Inc.' in x).sum())
- B. 1.counter = 0
2.
3.def count(x):
4. if 'Inc.' in x['supplier']:
5. counter = counter + 1
6.
7.itemsDf.foreach(count)
8.print(counter) - C. 1.accum=sc.accumulator(0)
2.
3.def check_if_inc_in_supplier(row):
4. if 'Inc.' in row['supplier']:
5. accum.add(1)
6.
7.itemsDf.foreach(check_if_inc_in_supplier)
8.print(accum.value) - D. print(itemsDf.foreach(lambda x: 'Inc.' in x))
- E. 1.counter = 0
2.
3.for index, row in itemsDf.iterrows():
4. if 'Inc.' in row['supplier']:
5. counter = counter + 1
6.
7.print(counter)
正解：C
解説：
Explanation
Correct code block:
accum=sc.accumulator(0)
def check_if_inc_in_supplier(row):
if 'Inc.' in row['supplier']:
accum.add(1)
itemsDf.foreach(check_if_inc_in_supplier)
print(accum.value)
To answer this question correctly, you need to know both about the DataFrame.foreach() method and accumulators.
When Spark runs the code, it executes it on the executors. The executors do not have any information about variables outside of their scope. This is whhy simply using a Python variable counter, like in the two examples that start with counter = 0, will not work. You need to tell the executors explicitly that counter is a special shared variable, an Accumulator, which is managed by the driver and can be accessed by all executors for the purpose of adding to it.
If you have used Pandas in the past, you might be familiar with the iterrows() command. Notice that there is no such command in PySpark.
The two examples that start with print do not work, since DataFrame.foreach() does not have a return value.
More info: pyspark.sql.DataFrame.foreach - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 # 85
The code block shown below should return a column that indicates through boolean variables whether rows in DataFrame transactionsDf have values greater or equal to 20 and smaller or equal to
30 in column storeId and have the value 2 in column productId. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__((__2__.__3__) __4__ (__5__))
- A. 1. where
2. col("storeId")
3. geq(20).leq(30)
4. &
5. col("productId")==2 - B. 1. select
2. col("storeId")
3. between(20, 30)
4. &
5. col("productId")==2 - C. 1. select
2. col("storeId")
3. between(20, 30)
4. &&
5. col("productId")=2 - D. 1. select
2. "storeId"
3. between(20, 30)
4. &&
5. col("productId")==2 - E. 1. select
2. col("storeId")
3. between(20, 30)
4. and
5. col("productId")==2
正解：C
解説：
Explanation
Correct code block:
transactionsDf.select((col("storeId").between(20, 30)) & (col("productId")==2)) Although this question may make you think that it asks for a filter or where statement, it does not. It asks explicity to return a column with booleans - this should point you to the select statement.
Another trick here is the rarely used between() method. It exists and resolves to ((storeId >= 20) AND (storeId
<= 30)) in SQL. geq() and leq() do not exist.
Another riddle here is how to chain the two conditions. The only valid answer here is &. Operators like && or and are not valid. Other boolean operators that would be valid in Spark are | and.
Static notebook | Dynamic notebook: See test 1
質問 # 86
......
合格保証付きのDatabricks Certification Associate-Developer-Apache-Spark試験問題集：https://jp.fast2test.com/Associate-Developer-Apache-Spark-premium-file.html
Associate-Developer-Apache-Spark練習テストエンジンで今すぐ使おう179試験問題：https://drive.google.com/open?id=1iXTRIAawpQAStgIN7QLWpHI0IyLvHG7q