最新のAssociate-Developer-Apache-Spark実際の無料試験問題更新された179問あります
無料で使えるAssociate-Developer-Apache-Spark試験ブレーン問題集認定ガイドの問題と解答
質問 80
Which of the following describes characteristics of the Spark UI?
- A. Via the Spark UI, workloads can be manually distributed across executors.
- B. Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL.
- C. There is a place in the Spark UI that shows the property spark.executor.memory.
- D. Via the Spark UI, stage execution speed can be modified.
- E. The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster.
正解: C
解説:
Explanation
There is a place in the Spark UI that shows the property spark.executor.memory.
Correct, you can see Spark properties such as spark.executor.memory in the Environment tab.
Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL.
Wrong - Jobs, Stages, Storage, Executors, and SQL are all tabs in the Spark UI. DAGs can be inspected in the
"Jobs" tab in the job details or in the Stages or SQL tab, but are not a separate tab.
Via the Spark UI, workloads can be manually distributed across distributors.
No, the Spark UI is meant for inspecting the inner workings of Spark which ultimately helps understand, debug, and optimize Spark transactions.
Via the Spark UI, stage execution speed can be modified.
No, see above.
The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster.
No, there is no Scheduler tab.
質問 81
The code block shown below should return a DataFrame with all columns of DataFrame transactionsDf, but only maximum 2 rows in which column productId has at least the value 2. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__).__3__
- A. 1. where
2. "productId" > 2
3. max(2) - B. 1. filter
2. col("productId") >= 2
3. limit(2) - C. 1. where
2. transactionsDf[productId] >= 2
3. limit(2) - D. 1. where
2. productId >= 2
3. limit(2) - E. 1. filter
2. productId > 2
3. max(2)
正解: B
解説:
Explanation
Correct code block:
transactionsDf.filter(col("productId") >= 2).limit(2)
The filter and where operators in gap 1 are just aliases of one another, so you cannot use them to pick the right answer.
The column definition in gap 2 is more helpful. The DataFrame.filter() method takes an argument of type Column or str. From all possible answers, only the one including col("productId") >= 2 fits this profile, since it returns a Column type.
The answer option using "productId" > 2 is invalid, since Spark does not understand that "productId" refers to column productId. The answer option using transactionsDf[productId] >= 2 is wrong because you cannot refer to a column using square bracket notation in Spark (if you are coming from Python using Pandas, this is something to watch out for). In all other options, productId is being referred to as a Python variable, so they are relatively easy to eliminate.
Also note that the question asks for the value in column productId being at least 2. This translates to a
"greater or equal" sign (>= 2), but not a "greater" sign (> 2).
Another thing worth noting is that there is no DataFrame.max() method. If you picked any option including this, you may be confusing it with the pyspark.sql.functions.max method. The correct method to limit the amount of rows is the DataFrame.limit() method.
More info:
- pyspark.sql.DataFrame.filter - PySpark 3.1.2 documentation
- pyspark.sql.DataFrame.limit - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 82
Which of the following describes Spark actions?
- A. Actions are Spark's way of modifying RDDs.
- B. Stage boundaries are commonly established by actions.
- C. The driver receives data upon request by actions.
- D. Writing data to disk is the primary purpose of actions.
- E. Actions are Spark's way of exchanging data between executors.
正解: C
解説:
Explanation
The driver receives data upon request by actions.
Correct! Actions trigger the distributed execution of tasks on executors which, upon task completion, transfer result data back to the driver.
Actions are Spark's way of exchanging data between executors.
No. In Spark, data is exchanged between executors via shuffles.
Writing data to disk is the primary purpose of actions.
No. The primary purpose of actions is to access data that is stored in Spark's RDDs and return the data, often in aggregated form, back to the driver.
Actions are Spark's way of modifying RDDs.
Incorrect. Firstly, RDDs are immutable - they cannot be modified. Secondly, Spark generates new RDDs via transformations and not actions.
Stage boundaries are commonly established by actions.
Wrong. A stage boundary is commonly established by a shuffle, for example caused by a wide transformation.
質問 83
The code block shown below should return a DataFrame with columns transactionsId, predError, value, and f from DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__)
- A. 1. select
2. "transactionId, predError, value, f" - B. 1. filter
2. "transactionId", "predError", "value", "f" - C. 1. select
2. col(["transactionId", "predError", "value", "f"]) - D. 1. where
2. col("transactionId"), col("predError"), col("value"), col("f") - E. 1. select
2. ["transactionId", "predError", "value", "f"]
正解: E
解説:
Explanation
Correct code block:
transactionsDf.select(["transactionId", "predError", "value", "f"])
The DataFrame.select returns specific columns from the DataFrame and accepts a list as its only argument.
Thus, this is the correct choice here. The option using col(["transactionId", "predError",
"value", "f"]) is invalid, since inside col(), one can only pass a single column name, not a list. Likewise, all columns being specified in a single string like "transactionId, predError, value, f" is not valid syntax.
filter and where filter rows based on conditions, they do not control which columns to return.
Static notebook | Dynamic notebook: See test 2
質問 84
The code block displayed below contains an error. The code block should configure Spark so that DataFrames up to a size of 20 MB will be broadcast to all worker nodes when performing a join.
Find the error.
Code block:
- A. The passed limit has the wrong variable type.
- B. spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 20)
- C. Spark will only broadcast DataFrames that are much smaller than the default value.
- D. The correct option to write configurations is through spark.config and not spark.conf.
- E. Spark will only apply the limit to threshold joins and not to other joins.
- F. The command is evaluated lazily and needs to be followed by an action.
正解: C
解説:
Explanation
This is question is hard. Let's assess the different answers one-by-one.
Spark will only broadcast DataFrames that are much smaller than the default value.
This is correct. The default value is 10 MB (10485760 bytes). Since the configuration for spark.sql.autoBroadcastJoinThreshold expects a number in bytes (and not megabytes), the code block sets the limits to merely 20 bytes, instead of the requested 20 * 1024 * 1024 (= 20971520) bytes.
The command is evaluated lazily and needs to be followed by an action.
No, this command is evaluated right away!
Spark will only apply the limit to threshold joins and not to other joins.
There are no "threshold joins", so this option does not make any sense.
The correct option to write configurations is through spark.config and not spark.conf.
No, it is indeed spark.conf!
The passed limit has the wrong variable type.
The configuration expects the number of bytes, a number, as an input. So, the 20 provided in the code block is fine.
質問 85
Which of the following code blocks reads JSON file imports.json into a DataFrame?
- A. spark.read().mode("json").path("/FileStore/imports.json")
- B. spark.read.format("json").path("/FileStore/imports.json")
- C. spark.read().json("/FileStore/imports.json")
- D. spark.read.json("/FileStore/imports.json")
- E. spark.read("json", "/FileStore/imports.json")
正解: D
解説:
Explanation
Static notebook | Dynamic notebook: See test 1
(https://flrs.github.io/spark_practice_tests_code/#1/25.html ,
https://bit.ly/sparkpracticeexams_import_instructions)
質問 86
Which of the following describes a narrow transformation?
- A. A narrow transformation is an operation in which no data is exchanged across the cluster.
- B. A narrow transformation is a process in which data from multiple RDDs is used.
- C. A narrow transformation is an operation in which data is exchanged across the cluster.
- D. A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like 16-bit or 8-bit float variables.
- E. narrow transformation is an operation in which data is exchanged across partitions.
正解: A
解説:
Explanation
A narrow transformation is an operation in which no data is exchanged across the cluster.
Correct! In narrow transformations, no data is exchanged across the cluster, since these transformations do not require any data from outside of the partition they are applied on. Typical narrow transformations include filter, drop, and coalesce.
A narrow transformation is an operation in which data is exchanged across partitions.
No, that would be one definition of a wide transformation, but not of a narrow transformation. Wide transformations typically cause a shuffle, in which data is exchanged across partitions, executors, and the cluster.
A narrow transformation is an operation in which data is exchanged across the cluster.
No, see explanation just above this one.
A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like
16-bit or 8-bit float variables.
No, type conversion has nothing to do with narrow transformations in Spark.
A narrow transformation is a process in which data from multiple RDDs is used.
No. A resilient distributed dataset (RDD) can be described as a collection of partitions. In a narrow transformation, no data is exchanged between partitions. Thus, no data is exchanged between RDDs.
One could say though that a narrow transformation and, in fact, any transformation results in a new RDD being created. This is because a transformation results in a change to an existing RDD (RDDs are the foundation of other Spark data structures, like DataFrames). But, since RDDs are immutable, a new RDD needs to be created to reflect the change caused by the transformation.
More info: Spark Transformation and Action: A Deep Dive | by Misbah Uddin | CodeX | Medium
質問 87
Which of the following statements about executors is correct, assuming that one can consider each of the JVMs working as executors as a pool of task execution slots?
- A. An executor runs on a single core.
- B. There must be more slots than tasks.
- C. Tasks run in parallel via slots.
- D. There must be less executors than tasks.
- E. Slot is another name for executor.
正解: C
解説:
Explanation
Tasks run in parallel via slots.
Correct. Given the assumption, an executor then has one or more "slots", defined by the equation spark.executor.cores / spark.task.cpus. With the executor's resources divided into slots, each task takes up a slot and multiple tasks can be executed in parallel.
Slot is another name for executor.
No, a slot is part of an executor.
An executor runs on a single core.
No, an executor can occupy multiple cores. This is set by the spark.executor.cores option.
There must be more slots than tasks.
No. Slots just process tasks. One could imagine a scenario where there was just a single slot for multiple tasks, processing one task at a time. Granted - this is the opposite of what Spark should be used for, which is distributed data processing over multiple cores and machines, performing many tasks in parallel.
There must be less executors than tasks.
No, there is no such requirement.
More info: Spark Architecture | Distributed Systems Architecture (https://bit.ly/3x4MZZt)
質問 88
The code block displayed below contains an error. The code block is intended to return all columns of DataFrame transactionsDf except for columns predError, productId, and value. Find the error.
Excerpt of DataFrame transactionsDf:
transactionsDf.select(~col("predError"), ~col("productId"), ~col("value"))
- A. The select operator should be replaced by the drop operator.
- B. The select operator should be replaced with the deselect operator.
- C. The column names in the select operator should not be strings and wrapped in the col operator, so they should be expressed like select(~col(predError), ~col(productId), ~col(value)).
- D. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value as strings.
(Correct) - E. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value wrapped in the col operator so they should be expressed like drop(col(predError), col(productId), col(value)).
正解: D
解説:
Explanation
Correct code block:
transactionsDf.drop("predError", "productId", "value")
Static notebook | Dynamic notebook: See test 1
質問 89
The code block shown below should return a DataFrame with only columns from DataFrame transactionsDf for which there is a corresponding transactionId in DataFrame itemsDf. DataFrame itemsDf is very small and much smaller than DataFrame transactionsDf. The query should be executed in an optimized way. Choose the answer that correctly fills the blanks in the code block to accomplish this.
__1__.__2__(__3__, __4__, __5__)
- A. 1. transactionsDf
2. join
3. broadcast(itemsDf)
4. "transactionId"
5. "left_semi" - B. 1. transactionsDf
2. join
3. broadcast(itemsDf)
4. transactionsDf.transactionId==itemsDf.transactionId
5. "outer" - C. 1. itemsDf
2. join
3. broadcast(transactionsDf)
4. "transactionId"
5. "left_semi" - D. 1. itemsDf
2. broadcast
3. transactionsDf
4. "transactionId"
5. "left_semi" - E. 1. transactionsDf
2. join
3. itemsDf
4. transactionsDf.transactionId==itemsDf.transactionId
5. "anti"
正解: A
解説:
Explanation
Correct code block:
transactionsDf.join(broadcast(itemsDf), "transactionId", "left_semi")
This question is extremely difficult and exceeds the difficulty of questions in the exam by far.
A first indication of what is asked from you here is the remark that "the query should be executed in an optimized way". You also have qualitative information about the size of itemsDf and transactionsDf. Given that itemsDf is "very small" and that the execution should be optimized, you should consider instructing Spark to perform a broadcast join, broadcasting the "very small" DataFrame itemsDf to all executors. You can explicitly suggest this to Spark via wrapping itemsDf into a broadcast() operator. One answer option does not include this operator, so you can disregard it. Another answer option wraps the broadcast() operator around transactionsDf - the bigger of the two DataFrames. This answer option does not make sense in the optimization context and can likewise be disregarded.
When thinking about the broadcast() operator, you may also remember that it is a method of pyspark.sql.functions. One answer option, however, resolves to itemsDf.broadcast([...]). The DataFrame class has no broadcast() method, so this answer option can be eliminated as well.
All two remaining answer options resolve to transactionsDf.join([...]) in the first 2 gaps, so you will have to figure out the details of the join now. You can pick between an outer and a left semi join. An outer join would include columns from both DataFrames, where a left semi join only includes columns from the "left" table, here transactionsDf, just as asked for by the question. So, the correct answer is the one that uses the left_semi join.
質問 90
Which of the following code blocks returns approximately 1000 rows, some of them potentially being duplicates, from the 2000-row DataFrame transactionsDf that only has unique rows?
- A. transactionsDf.sample(True, 0.5, force=True)
- B. transactionsDf.take(1000).distinct()
- C. transactionsDf.sample(True, 0.5)
- D. transactionsDf.sample(False, 0.5)
- E. transactionsDf.take(1000)
正解: C
解説:
Explanation
To solve this question, you need to know that DataFrame.sample() is not guaranteed to return the exact fraction of the number of rows specified as an argument. Furthermore, since duplicates may be returned, you should understand that the operator's withReplacement argument should be set to True. A force= argument for the operator does not exist.
While the take argument returns an exact number of rows, it will just take the first specified number of rows (1000 in this question) from the DataFrame. Since the DataFrame does not include duplicate rows, there is no potential of any of those returned rows being duplicates when using take(), so the correct answer cannot involve take().
More info: pyspark.sql.DataFrame.sample - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
質問 91
The code block displayed below contains multiple errors. The code block should remove column transactionDate from DataFrame transactionsDf and add a column transactionTimestamp in which dates that are expressed as strings in column transactionDate of DataFrame transactionsDf are converted into unix timestamps. Find the errors.
Sample of DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+----------------+
2.|transactionId|predError|value|storeId|productId| f| transactionDate|
3.+-------------+---------+-----+-------+---------+----+----------------+
4.| 1| 3| 4| 25| 1|null|2020-04-26 15:35|
5.| 2| 6| 7| 2| 2|null|2020-04-13 22:01|
6.| 3| 3| null| 25| 3|null|2020-04-02 10:53|
7.+-------------+---------+-----+-------+---------+----+----------------+ Code block:
1.transactionsDf = transactionsDf.drop("transactionDate")
2.transactionsDf["transactionTimestamp"] = unix_timestamp("transactionDate", "yyyy-MM-dd")
- A. Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment. Operator to_unixtime() should be used instead of unix_timestamp().
- B. Column transactionDate should be wrapped in a col() operator.
- C. Column transactionDate should be dropped after transactionTimestamp has been written. The withColumn operator should be used instead of the existing column assignment. Column transactionDate should be wrapped in a col() operator.
- D. The string indicating the date format should be adjusted. The withColumnReplaced operator should be used instead of the drop and assign pattern in the code block to replace column transactionDate with the new column transactionTimestamp.
- E. Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment.
正解: E
解説:
Explanation
This question requires a lot of thinking to get right. For solving it, you may take advantage of the digital notepad that is provided to you during the test. You have probably seen that the code block includes multiple errors. In the test, you are usually confronted with a code block that only contains a single error. However, since you are practicing here, this challenging multi-error question will make it easier for you to deal with single-error questions in the real exam.
You can clearly see that column transactionDate should be dropped only after transactionTimestamp has been written. This is because to generate column transactionTimestamp, Spark needs to read the values from column transactionDate.
Values in column transactionDate in the original transactionsDf DataFrame look like 2020-04-26 15:35. So, to convert those correctly, you would have to pass yyyy-MM-dd HH:mm. In other words:
The string indicating the date format should be adjusted.
While you might be tempted to change unix_timestamp() to to_unixtime() (in line with the from_unixtime() operator), this function does not exist in Spark. unix_timestamp() is the correct operator to use here.
Also, there is no DataFrame.withColumnReplaced() operator. A similar operator that exists is DataFrame.withColumnRenamed().
Whether you use col() or not is irrelevant with unix_timestamp() - the command is fine with both.
Finally, you cannot assign a column like transactionsDf["columnName"] = ... in Spark. This is Pandas syntax (Pandas is a popular Python package for data analysis), but it is not supported in Spark.
So, you need to use Spark's DataFrame.withColumn() syntax instead.
More info: pyspark.sql.functions.unix_timestamp - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
質問 92
The code block shown below should return only the average prediction error (column predError) of a random subset, without replacement, of approximately 15% of rows in DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__, __3__).__4__(avg('predError'))
- A. 1. fraction
2. 0.15
3. True
4. where - B. 1. fraction
2. False
3. 0.85
4. select - C. 1. sample
2. True
3. 0.15
4. filter - D. 1. sample
2. 0.85
3. False
4. select - E. 1. sample
2. False
3. 0.15
4. select
正解: E
解説:
Explanation
Correct code block:
transactionsDf.sample(withReplacement=False, fraction=0.15).select(avg('predError')) You should remember that getting a random subset of rows means sampling. This, in turn should point you to the DataFrame.sample() method. Once you know this, you can look up the correct order of arguments in the documentation (link below).
Lastly, you have to decide whether to use filter, where or select. where is just an alias for filter(). filter() is not the correct method to use here, since it would only allow you to filter rows based on some condition. However, the question asks to return only the average prediction error. You can control the columns that a query returns with the select() method - so this is the correct method to use here.
More info: pyspark.sql.DataFrame.sample - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
質問 93
Which of the following code blocks returns a DataFrame with an added column to DataFrame transactionsDf that shows the unix epoch timestamps in column transactionDate as strings in the format month/day/year in column transactionDateFormatted?
Excerpt of DataFrame transactionsDf:
- A. transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate"))
- B. transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="MM/dd/yyyy"))
- C. transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="dd/MM/yyyy"))
- D. transactionsDf.apply(from_unixtime(format="MM/dd/yyyy")).asColumn("transactionDateFormatted")
- E. transactionsDf.withColumnRenamed("transactionDate", "transactionDateFormatted", from_unixtime("transactionDateFormatted", format="MM/dd/yyyy"))
正解: B
解説:
Explanation
transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="MM/dd/yyyy")) Correct. This code block adds a new column with the name transactionDateFormatted to DataFrame transactionsDf, using Spark's from_unixtime method to transform values in column transactionDate into strings, following the format requested in the question.
transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="dd/MM/yyyy")) No. Although almost correct, this uses the wrong format for the timestamp to date conversion: day/month/year instead of month/day/year.
transactionsDf.withColumnRenamed("transactionDate", "transactionDateFormatted", from_unixtime("transactionDateFormatted", format="MM/dd/yyyy")) Incorrect. This answer uses wrong syntax. The command DataFrame.withColumnRenamed() is for renaming an existing column only has two string parameters, specifying the old and the new name of the column.
transactionsDf.apply(from_unixtime(format="MM/dd/yyyy")).asColumn("transactionDateFormatted") Wrong. Although this answer looks very tempting, it is actually incorrect Spark syntax. In Spark, there is no method DataFrame.apply(). Spark has an apply() method that can be used on grouped data - but this is irrelevant for this question, since we do not deal with grouped data here.
transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate")) No. Although this is valid Spark syntax, the strings in column transactionDateFormatted would look like this:
2020-04-26 15:35:32, the default format specified in Spark for from_unixtime and not what is asked for in the question.
More info: pyspark.sql.functions.from_unixtime - PySpark 3.1.1 documentation and pyspark.sql.DataFrame.withColumnRenamed - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 94
Which of the following code blocks returns a new DataFrame with the same columns as DataFrame transactionsDf, except for columns predError and value which should be removed?
- A. transactionsDf.drop("predError", "value")
- B. transactionsDf.drop(predError, value)
- C. transactionsDf.drop(col("predError"), col("value"))
- D. transactionsDf.drop("predError & value")
- E. transactionsDf.drop(["predError", "value"])
正解: A
解説:
Explanation
More info: pyspark.sql.DataFrame.drop - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
質問 95
......
Associate-Developer-Apache-Spark認定概要最新のAssociate-Developer-Apache-SparkのPDF問題集：https://jp.fast2test.com/Associate-Developer-Apache-Spark-premium-file.html
トップクラスDatabricks Associate-Developer-Apache-Spark試験材料で学習ガイド！練習問題バージョン：https://drive.google.com/open?id=1iXTRIAawpQAStgIN7QLWpHI0IyLvHG7q