Databricksは2021年最新のAssociate-Developer-Apache-Sparkテスト解説（更新されたのは179問があります) [Q85-Q103]

Databricksは2021年最新のAssociate-Developer-Apache-Sparkテスト解説（更新されたのは179問があります)

Associate-Developer-Apache-Spark試験問題集を提供していますDatabricks問題

質問 85
Which of the following code blocks displays various aggregated statistics of all columns in DataFrame transactionsDf, including the standard deviation and minimum of values in each column?

A. transactionsDf.summary()
B. transactionsDf.agg("count", "mean", "stddev", "25%", "50%", "75%", "min")
C. transactionsDf.summary().show()
D. transactionsDf.summary("count", "mean", "stddev", "25%", "50%", "75%", "max").show()
E. transactionsDf.agg("count", "mean", "stddev", "25%", "50%", "75%", "min").show()

正解: C

解説:
Explanation
The DataFrame.summary() command is very practical for quickly calculating statistics of a DataFrame. You need to call .show() to display the results of the calculation. By default, the command calculates various statistics (see documentation linked below), including standard deviation and minimum.
Note that the answer that lists many options in the summary() parentheses does not include the minimum, which is asked for in the question.
Answer options that include agg() do not work here as shown, since DataFrame.agg() expects more complex, column-specific instructions on how to aggregate values.
More info:
- pyspark.sql.DataFrame.summary - PySpark 3.1.2 documentation
- pyspark.sql.DataFrame.agg - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

質問 86
Which of the following code blocks returns DataFrame transactionsDf sorted in descending order by column predError, showing missing values last?

A. transactionsDf.orderBy("predError").asc_nulls_last()
B. transactionsDf.desc_nulls_last("predError")
C. transactionsDf.orderBy("predError").desc_nulls_last()
D. transactionsDf.sort(asc_nulls_last("predError"))
E. transactionsDf.sort("predError", ascending=False)

正解: E

解説:
Explanation
transactionsDf.sort("predError", ascending=False)
Correct! When using DataFrame.sort() and setting ascending=False, the DataFrame will be sorted by the specified column in descending order, putting all missing values last. An alternative, although not listed as an answer here, would be transactionsDf.sort(desc_nulls_last("predError")).
transactionsDf.sort(asc_nulls_last("predError"))
Incorrect. While this is valid syntax, the DataFrame will be sorted on column predError in ascending order and not in descending order, putting missing values last.
transactionsDf.desc_nulls_last("predError")
Wrong, this is invalid syntax. There is no method DataFrame.desc_nulls_last() in the Spark API. There is a Spark function desc_nulls_last() however (link see below).
transactionsDf.orderBy("predError").desc_nulls_last()
No. While transactionsDf.orderBy("predError") is correct syntax (although it sorts the DataFrame by column predError in ascending order) and returns a DataFrame, there is no method DataFrame.desc_nulls_last() in the Spark API. There is a Spark function desc_nulls_last() however (link see below).
transactionsDf.orderBy("predError").asc_nulls_last()
Incorrect. There is no method DataFrame.asc_nulls_last() in the Spark API (see above).
More info: pyspark.sql.functions.desc_nulls_last - PySpark 3.1.2 documentation and pyspark.sql.DataFrame.sort - PySpark 3.1.2 documentation (https://bit.ly/3g1JtbI , https://bit.ly/2R90NCS) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/32.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

質問 87
Which of the following code blocks performs an inner join of DataFrames transactionsDf and itemsDf on columns productId and itemId, respectively, excluding columns value and storeId from DataFrame transactionsDf and column attributes from DataFrame itemsDf?

A. 1.transactionsDf.createOrReplaceTempView('transactionsDf')
2.itemsDf.createOrReplaceTempView('itemsDf')
3.
4.statement = """
5.SELECT * FROM transactionsDf
6.INNER JOIN itemsDf
7.ON transactionsDf.productId==itemsDf.itemId
8."""
9.spark.sql(statement).drop("value", "storeId", "attributes")
B. 1.transactionsDf \
2. .drop(col('value'), col('storeId')) \
3. .join(itemsDf.drop(col('attributes')), col('productId')==col('itemId'))
C. transactionsDf.drop("value", "storeId").join(itemsDf.drop("attributes"),
"transactionsDf.productId==itemsDf.itemId")
D. 1.transactionsDf.createOrReplaceTempView('transactionsDf')
2.itemsDf.createOrReplaceTempView('itemsDf')
3.
4.spark.sql("SELECT -value, -storeId FROM transactionsDf INNER JOIN itemsDf ON productId==itemId").drop("attributes")
E. transactionsDf.drop('value', 'storeId').join(itemsDf.select('attributes'), transactionsDf.productId==itemsDf.itemId)

正解: A

解説:
Explanation
This question offers you a wide variety of answers for a seemingly simple question. However, this variety reflects the variety of ways that one can express a join in PySpark. You need to understand some SQL syntax to get to the correct answer here.
transactionsDf.createOrReplaceTempView('transactionsDf')
itemsDf.createOrReplaceTempView('itemsDf')
statement = """
SELECT * FROM transactionsDf
INNER JOIN itemsDf
ON transactionsDf.productId==itemsDf.itemId
"""
spark.sql(statement).drop("value", "storeId", "attributes")
Correct - this answer uses SQL correctly to perform the inner join and afterwards drops the unwanted columns. This is totally fine. If you are unfamiliar with the triple-quote """ in Python: This allows you to express strings as multiple lines.
transactionsDf \
drop(col('value'), col('storeId')) \
join(itemsDf.drop(col('attributes')), col('productId')==col('itemId'))
No, this answer option is a trap, since DataFrame.drop() does not accept a list of Column objects. You could use transactionsDf.drop('value', 'storeId') instead.
transactionsDf.drop("value", "storeId").join(itemsDf.drop("attributes"),
"transactionsDf.productId==itemsDf.itemId")
Incorrect - Spark does not evaluate "transactionsDf.productId==itemsDf.itemId" as a valid join expression.
This would work if it would not be a string.
transactionsDf.drop('value', 'storeId').join(itemsDf.select('attributes'), transactionsDf.productId==itemsDf.itemId) Wrong, this statement incorrectly uses itemsDf.select instead of itemsDf.drop.
transactionsDf.createOrReplaceTempView('transactionsDf')
itemsDf.createOrReplaceTempView('itemsDf')
spark.sql("SELECT -value, -storeId FROM transactionsDf INNER JOIN itemsDf ON productId==itemId").drop("attributes") No, here the SQL expression syntax is incorrect. Simply specifying -columnName does not drop a column.
More info: pyspark.sql.DataFrame.join - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

質問 88
The code block displayed below contains at least one error. The code block should return a DataFrame with only one column, result. That column should include all values in column value from DataFrame transactionsDf raised to the power of 5, and a null value for rows in which there is no value in column value. Find the error(s).
Code block:
1.from pyspark.sql.functions import udf
2.from pyspark.sql import types as T
3.
4.transactionsDf.createOrReplaceTempView('transactions')
5.
6.def pow_5(x):
7. return x**5
8.
9.spark.udf.register(pow_5, 'power_5_udf', T.LongType())
10.spark.sql('SELECT power_5_udf(value) FROM transactions')

A. The pow_5 method is unable to handle empty values in column value and the name of the column in the returned DataFrame is not result.
B. The returned DataFrame includes multiple columns instead of just one column.
C. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and the SparkSession cannot access the transactionsDf DataFrame.
D. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.
E. The pow_5 method is unable to handle empty values in column value, the UDF function is not registered properly with the Spark driver, and the name of the column in the returned DataFrame is not result.

正解: D

解説:
Explanation
Correct code block:
from pyspark.sql.functions import udf
from pyspark.sql import types as T
transactionsDf.createOrReplaceTempView('transactions')
def pow_5(x):
if x:
return x**5
return x
spark.udf.register('power_5_udf', pow_5, T.LongType())
spark.sql('SELECT power_5_udf(value) AS result FROM transactions')
Here it is important to understand how the pow_5 method handles empty values. In the wrong code block above, the pow_5 method is unable to handle empty values and will throw an error, since Python's ** operator cannot deal with any null value Spark passes into method pow_5.
The order of arguments for registering the UDF function with Spark via spark.udf.register matters. In the code snippet in the question, the arguments for the SQL method name and the actual Python function are switched. You can read more about the arguments of spark.udf.register and see some examples of its usage in the documentation (link below).
Finally, you should recognize that in the original code block, an expression to rename column created through the UDF function is missing. The renaming is done by SQL's AS result argument.
Omitting that argument, you end up with the column name power_5_udf(value) and not result.
More info: pyspark.sql.functions.udf - PySpark 3.1.1 documentation

質問 89
Which of the following describes slots?

A. A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.
B. Slots are dynamically created and destroyed in accordance with an executor's workload.
C. A slot is always limited to a single core.
Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.
D. To optimize I/O performance, Spark stores data on disk in multiple slots.

正解: A

解説:
Explanation
Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.
Wrong, executors communicate with the driver directly.
Slots are dynamically created and destroyed in accordance with an executor's workload.
No, Spark does not actively create and destroy slots in accordance with the workload. Per executor, slots are made available in accordance with how many cores per executor (property spark.executor.cores) and how many CPUs per task (property spark.task.cpus) the Spark configuration calls for.
A slot is always limited to a single core.
No, a slot can span multiple cores. If a task would require multiple cores, it would have to be executed through a slot that spans multiple cores.
In Spark documentation, "core" is often used interchangeably with "thread", although "thread" is the more accurate word. A single physical core may be able to make multiple threads available. So, it is better to say that a slot can span multiple threads.
To optimize I/O performance, Spark stores data on disk in multiple slots.
No - Spark stores data on disk in multiple partitions, not slots.
More info: Spark Architecture | Distributed Systems Architecture

質問 90
Which of the following code blocks returns only rows from DataFrame transactionsDf in which values in column productId are unique?

A. transactionsDf.distinct("productId")
B. transactionsDf.drop_duplicates(subset="productId")
C. transactionsDf.dropDuplicates(subset=["productId"])
D. transactionsDf.dropDuplicates(subset="productId")
E. transactionsDf.unique("productId")

正解: C

解説:
Explanation
Although the question suggests using a method called unique() here, that method does not actually exist in PySpark. In PySpark, it is called distinct(). But then, this method is not the right one to use here, since with distinct() we could filter out unique values in a specific column.
However, we want to return the entire rows here. So the trick is to use dropDuplicates with the subset keyword parameter. In the documentation for dropDuplicates, the examples show that subset should be used with a list. And this is exactly the key to solving this question: The productId column needs to be fed into the subset argument in a list, even though it is just a single column.
More info: pyspark.sql.DataFrame.dropDuplicates - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

質問 91
The code block shown below should read all files with the file ending .png in directory path into Spark.
Choose the answer that correctly fills the blanks in the code block to accomplish this.
spark.__1__.__2__(__3__).option(__4__, "*.png").__5__(path)

A. 1. open
2. format
3. "image"
4. "fileType"
5. open
B. 1. read()
2. format
3. "binaryFile"
4. "recursiveFileLookup"
5. load
C. 1. read
2. format
3. binaryFile
4. pathGlobFilter
5. load
D. 1. open
2. as
3. "binaryFile"
4. "pathGlobFilter"
5. load
E. 1. read
2. format
3. "binaryFile"
4. "pathGlobFilter"
5. load

正解: E

解説:
Explanation
Correct code block:
spark.read.format("binaryFile").option("recursiveFileLookup", "*.png").load(path) Spark can deal with binary files, like images. Using the binaryFile format specification in the SparkSession's read API is the way to read in those files. Remember that, to access the read API, you need to start the command with spark.read. The pathGlobFilter option is a great way to filter files by name (and ending). Finally, the path can be specified using the load operator - the open operator shown in one of the answers does not exist.

質問 92
The code block displayed below contains an error. The code block should arrange the rows of DataFrame transactionsDf using information from two columns in an ordered fashion, arranging first by column value, showing smaller numbers at the top and greater numbers at the bottom, and then by column predError, for which all values should be arranged in the inverse way of the order of items in column value. Find the error.
Code block:
transactionsDf.orderBy('value', asc_nulls_first(col('predError')))

A. Column predError should be sorted by desc_nulls_first() instead.
B. Instead of orderBy, sort should be used.
C. Column predError should be sorted in a descending way, putting nulls last.
D. Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
E. Column value should be wrapped by the col() operator.

正解: C

解説:
Explanation
Correct code block:
transactionsDf.orderBy('value', desc_nulls_last('predError'))
Column predError should be sorted in a descending way, putting nulls last.
Correct! By default, Spark sorts ascending, putting nulls first. So, the inverse sort of the default sort is indeed desc_nulls_last.
Instead of orderBy, sort should be used.
No. DataFrame.sort() orders data per partition, it does not guarantee a global order. This is why orderBy is the more appropriate operator here.
Column value should be wrapped by the col() operator.
Incorrect. DataFrame.sort() accepts both string and Column objects.
Column predError should be sorted by desc_nulls_first() instead.
Wrong. Since Spark's default sort order matches asc_nulls_first(), nulls would have to come last when inverted.
Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
No, this would just sort the DataFrame by the very last column, but would not take information from both columns into account, as noted in the question.
More info: pyspark.sql.DataFrame.orderBy - PySpark 3.1.2 documentation, pyspark.sql.functions.desc_nulls_last - PySpark 3.1.2 documentation, sort() vs orderBy() in Spark | Towards Data Science Static notebook | Dynamic notebook: See test 3

質問 93
Which of the following describes properties of a shuffle?

A. A shuffle is one of many actions in Spark.
B. Shuffles belong to a class known as "full transformations".
C. In a shuffle, Spark writes data to disk.
D. Operations involving shuffles are never evaluated lazily.
E. Shuffles involve only single partitions.

正解: C

解説:
Explanation
In a shuffle, Spark writes data to disk.
Correct! Spark's architecture dictates that intermediate results during a shuffle are written to disk.
A shuffle is one of many actions in Spark.
Incorrect. A shuffle is a transformation, but not an action.
Shuffles involve only single partitions.
No, shuffles involve multiple partitions. During a shuffle, Spark generates output partitions from multiple input partitions.
Operations involving shuffles are never evaluated lazily.
Wrong. A shuffle is a costly operation and Spark will evaluate it as lazily as other transformations. This is, until a subsequent action triggers its evaluation.
Shuffles belong to a class known as "full transformations".
Not quite. Shuffles belong to a class known as "wide transformations". "Full transformation" is not a relevant term in Spark.
More info: Spark - The Definitive Guide, Chapter 2 and Spark: disk I/O on stage boundaries explanation - Stack Overflow

質問 94
The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.
Code block:
transactionsDf.coalesce(14, ("storeId", "transactionDate"))

A. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
(Correct)
B. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.
C. Operator coalesce needs to be replaced by repartition.
D. Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.
E. The parentheses around the column names need to be removed and .select() needs to be appended to the code block.

正解: A

解説:
Explanation
Correct code block:
transactionsDf.repartition(14, "storeId", "transactionDate").count()
Since we do not know how many partitions DataFrame transactionsDf has, we cannot safely use coalesce, since it would not make any change if the current number of partitions is smaller than 14.
So, we need to use repartition.
In the Spark documentation, the call structure for repartition is shown like this:
DataFrame.repartition(numPartitions, *cols). The * operator means that any argument after numPartitions will be interpreted as column. Therefore, the brackets need to be removed.
Finally, the question specifies that after the execution the DataFrame should be divided. So, indirectly this question is asking us to append an action to the code block. Since .select() is a transformation. the only possible choice here is .count().
More info: pyspark.sql.DataFrame.repartition - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

質問 95
Which of the following code blocks performs a join in which the small DataFrame transactionsDf is sent to all executors where it is joined with DataFrame itemsDf on columns storeId and itemId, respectively?

A. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "right_outer")
B. itemsDf.merge(transactionsDf, "itemsDf.itemId == transactionsDf.storeId", "broadcast")
C. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "broadcast")
D. itemsDf.join(broadcast(transactionsDf), itemsDf.itemId == transactionsDf.storeId)
E. itemsDf.join(transactionsDf, broadcast(itemsDf.itemId == transactionsDf.storeId))

正解: D

解説:
Explanation
The issue with all answers that have "broadcast" as very last argument is that "broadcast" is not a valid join type. While the entry with "right_outer" is a valid statement, it is not a broadcast join. The item where broadcast() is wrapped around the equality condition is not valid code in Spark. broadcast() needs to be wrapped around the name of the small DataFrame that should be broadcast.
More info: Learning Spark, 2nd Edition, Chapter 7
Static notebook | Dynamic notebook: See test 1
tion and explanation?

質問 96
The code block shown below should write DataFrame transactionsDf to disk at path csvPath as a single CSV file, using tabs (\t characters) as separators between columns, expressing missing values as string n/a, and omitting a header row with column names. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__.write.__2__(__3__, " ").__4__.__5__(csvPath)

A. 1. coalesce(1)
2. option
3. "colsep"
4. option("nullValue", "n/a")
5. path
B. 1. coalesce(1)
2. option
3. "sep"
4. option("header", True)
5. path
C. 1. csv
2. option
3. "sep"
4. option("emptyValue", "n/a")
5. path
* 1. repartition(1)
2. mode
3. "sep"
4. mode("nullValue", "n/a")
5. csv
D. 1. repartition(1)
2. option
3. "sep"
4. option("nullValue", "n/a")
5. csv
(Correct)

正解: D

解説:
Explanation
Correct code block:
transactionsDf.repartition(1).write.option("sep", "\t").option("nullValue", "n/a").csv(csvPath) It is important here to understand that the question specifically asks for writing the DataFrame as a single CSV file. This should trigger you to think about partitions. By default, every partition is written as a separate file, so you need to include repatition(1) into your call. coalesce(1) works here, too!
Secondly, the question is very much an invitation to search through the parameters in the Spark documentation that work with DataFrameWriter.csv (link below). You will also need to know that you need an option() statement to apply these parameters.
The final concern is about the general call structure. Once you have called accessed write of your DataFrame, options follow and then you write the DataFrame with csv. Instead of csv(csvPath), you could also use save(csvPath, format='csv') here.
More info: pyspark.sql.DataFrameWriter.csv - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

質問 97
Which of the following code blocks stores DataFrame itemsDf in executor memory and, if insufficient memory is available, serializes it and saves it to disk?

A. itemsDf.cache()
B. itemsDf.write.option('destination', 'memory').save()
C. itemsDf.store()
D. itemsDf.cache(StorageLevel.MEMORY_AND_DISK)
E. itemsDf.persist(StorageLevel.MEMORY_ONLY)

正解: A

解説:
Explanation
The key to solving this question is knowing (or reading in the documentation) that, by default, cache() stores values to memory and writes any partitions for which there is insufficient memory to disk. persist() can achieve the exact same behavior, however not with the StorageLevel.MEMORY_ONLY option listed here. It is also worth noting that cache() does not have any arguments.
If you have troubles finding the storage level information in the documentation, please also see this student Q&A thread that sheds some light here.
Static notebook | Dynamic notebook: See test 2

A. itemsDf.explode(attributes).alias("attributes_exploded").filter(col("attributes_exploded").contains("i"))
B. itemsDf.select(explode("attributes")).filter("attributes_exploded".contains("i"))
C. itemsDf.select(col("attributes").explode().alias("attributes_exploded")).filter(col("attributes_exploded").co
D. itemsDf.select(explode("attributes").alias("attributes_exploded")).filter(col("attributes_exploded").contain
E. itemsDf.select(explode("attributes").alias("attributes_exploded")).filter(attributes_exploded.contains("i"))

正解: D

解説:
Explanation
Result of correct code block:
+-------------------+
|attributes_exploded|
+-------------------+
| winter|
| cooling|
+-------------------+
To solve this question, you need to know about explode(). This operation helps you to split up arrays into single rows. If you did not have a chance to familiarize yourself with this method yet, find more examples in the documentation (link below).
Note that explode() is a method made available through pyspark.sql.functions - it is not available as a method of a DataFrame or a Column, as written in some of the answer options.
More info: pyspark.sql.functions.explode - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2

質問 99
Which of the following code blocks reorders the values inside the arrays in column attributes of DataFrame itemsDf from last to first one in the alphabet?
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+

A. itemsDf.withColumn('attributes', sort(col('attributes'), asc=False))
B. itemsDf.withColumn('attributes', sort_array(desc('attributes')))
C. itemsDf.withColumn("attributes", sort_array("attributes", asc=False))
D. itemsDf.select(sort_array("attributes"))
E. itemsDf.withColumn('attributes', sort_array(col('attributes').desc()))

正解: C

解説:
Explanation
Output of correct code block:
+------+-----------------------------+-------------------+
|itemId|attributes |supplier |
+------+-----------------------------+-------------------+
|1 |[winter, cozy, blue] |Sports Company Inc.|
|2 |[summer, red, fresh, cooling]|YetiX |
|3 |[travel, summer, green] |Sports Company Inc.|
+------+-----------------------------+-------------------+
It can be confusing to differentiate between the different sorting functions in PySpark. In this case, a particularity about sort_array has to be considered: The sort direction is given by the second argument, not by the desc method. Luckily, this is documented in the documentation (link below). Also, for solving this question you need to understand the difference between sort and sort_array. With sort, you cannot sort values in arrays. Also, sort is a method of DataFrame, while sort_array is a method of pyspark.sql.functions.
More info: pyspark.sql.functions.sort_array - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2

質問 100
The code block displayed below contains an error. The code block should configure Spark to split data in 20 parts when exchanging data between executors for joins or aggregations. Find the error.
Code block:
spark.conf.set(spark.sql.shuffle.partitions, 20)

A. The code block expresses the option incorrectly.
B. The code block is missing a parameter.
C. The code block sets the wrong option.
D. The code block sets the incorrect number of parts.
E. The code block uses the wrong command for setting an option.

正解: A

解説:
Explanation
Correct code block:
spark.conf.set("spark.sql.shuffle.partitions", 20)
The code block expresses the option incorrectly.
Correct! The option should be expressed as a string.
The code block sets the wrong option.
No, spark.sql.shuffle.partitions is the correct option for the use case in the question.
The code block sets the incorrect number of parts.
Wrong, the code block correctly states 20 parts.
The code block uses the wrong command for setting an option.
No, in PySpark spark.conf.set() is the correct command for setting an option.
The code block is missing a parameter.
Incorrect, spark.conf.set() takes two parameters.
More info: Configuration - Spark 3.1.2 Documentation

質問 101
The code block displayed below contains an error. The code block should use Python method find_most_freq_letter to find the letter present most in column itemName of DataFrame itemsDf and return it in a new column most_frequent_letter. Find the error.
Code block:
1. find_most_freq_letter_udf = udf(find_most_freq_letter)
2. itemsDf.withColumn("most_frequent_letter", find_most_freq_letter("itemName"))

A. Spark is not using the UDF method correctly.
B. The "itemName" expression should be wrapped in col().
C. Spark is not adding a column.
D. UDFs do not exist in PySpark.
E. The UDF method is not registered correctly, since the return type is missing.

正解: A

解説:
Explanation
Correct code block:
find_most_freq_letter_udf = udf(find_most_frequent_letter)
itemsDf.withColumn("most_frequent_letter", find_most_freq_letter_udf("itemName")) Spark should use the previously registered find_most_freq_letter_udf method here - but it is not doing that in the original codeblock. There, it just uses the non-UDF version of the Python method.
Note that typically, we would have to specify a return type for udf(). Except in this case, since the default return type for udf() is a string which is what we are expecting here. If we wanted to return an integer variable instead, we would have to register the Python function as UDF using find_most_freq_letter_udf = udf(find_most_freq_letter, IntegerType()).
More info: pyspark.sql.functions.udf - PySpark 3.1.1 documentation

質問 102
Which of the following code blocks returns a DataFrame where columns predError and productId are removed from DataFrame transactionsDf?
Sample of DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId|f |
3.+-------------+---------+-----+-------+---------+----+
4.|1 |3 |4 |25 |1 |null|
5.|2 |6 |7 |2 |2 |null|
6.|3 |3 |null |25 |3 |null|
7.+-------------+---------+-----+-------+---------+----+

A. transactionsDf.drop("predError", "productId", "associateId")
B. transactionsDf.withColumnRemoved("predError", "productId")
C. transactionsDf.drop(col("predError", "productId"))
D. transactionsDf.dropColumns("predError", "productId", "associateId")
E. transactionsDf.drop(["predError", "productId", "associateId"])

正解: D

解説:
Explanation
The key here is to understand that columns that are passed to DataFrame.drop() are ignored if they do not exist in the DataFrame. So, passing column name associateId to transactionsDf.drop() does not have any effect.
Passing a list to transactionsDf.drop() is not valid. The documentation (link below) shows the call structure as DataFrame.drop(*cols). The * means that all arguments that are passed to DataFrame.drop() are read as columns. However, since a list of columns, for example ["predError",
"productId", "associateId"] is not a column, Spark will run into an error.
More info: pyspark.sql.DataFrame.drop - PySpark 3.1.1 documentation
Static notebook | Dynamic notebook: See test 1

質問 103
......

Associate-Developer-Apache-Spark認定ガイドPDFは100%カバー率でリアル試験問題：https://jp.fast2test.com/Associate-Developer-Apache-Spark-premium-file.html

合格させるAssociate-Developer-Apache-Sparkレビューガイド、信頼され続けるAssociate-Developer-Apache-Sparkテストエンジン：https://drive.google.com/open?id=1iXTRIAawpQAStgIN7QLWpHI0IyLvHG7q

関するブログ

もっと

Databricks認証 Associate-Developer-Apache-Spark試験問題集

弊社を連絡する

我々は１２時間以内ですべてのお問い合わせを答えます。

我々の働いている時間： ( GMT 0:00-15:00 )
月曜日から土曜日まで

サポート： 現在連絡

English Deutsch 繁体中文 한국어

Databricksは2021年最新のAssociate-Developer-Apache-Sparkテスト解説（更新されたのは179問があります) [Q85-Q103]

関するブログ

弊社を連絡する

フレンドリーリンク

トップ試験