Fast2test Associate-Developer-Apache-Spark問題集179問でDatabricks Certificationを確実実践
リアル最新Associate-Developer-Apache-Spark試験問題Associate-Developer-Apache-Spark問題集
質問 50
The code block shown below should return an exact copy of DataFrame transactionsDf that does not include rows in which values in column storeId have the value 25. Choose the answer that correctly fills the blanks in the code block to accomplish this.
- A. transactionsDf.remove(transactionsDf.storeId==25)
- B. transactionsDf.where(transactionsDf.storeId!=25)
- C. transactionsDf.filter(transactionsDf.storeId==25)
- D. transactionsDf.select(transactionsDf.storeId!=25)
- E. transactionsDf.drop(transactionsDf.storeId==25)
正解: B
解説:
Explanation
transactionsDf.where(transactionsDf.storeId!=25)
Correct. DataFrame.where() is an alias for the DataFrame.filter() method. Using this method, it is straightforward to filter out rows that do not have value 25 in column storeId.
transactionsDf.select(transactionsDf.storeId!=25)
Wrong. The select operator allows you to build DataFrames column-wise, but when using it as shown, it does not filter out rows.
transactionsDf.filter(transactionsDf.storeId==25)
Incorrect. Although the filter expression works for filtering rows, the == in the filtering condition is inappropriate. It should be != instead.
transactionsDf.drop(transactionsDf.storeId==25)
No. DataFrame.drop() is used to remove specific columns, but not rows, from the DataFrame.
transactionsDf.remove(transactionsDf.storeId==25)
False. There is no DataFrame.remove() operator in PySpark.
More info: pyspark.sql.DataFrame.where - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 51
Which of the following code blocks stores DataFrame itemsDf in executor memory and, if insufficient memory is available, serializes it and saves it to disk?
- A. itemsDf.persist(StorageLevel.MEMORY_ONLY)
- B. itemsDf.write.option('destination', 'memory').save()
- C. itemsDf.cache(StorageLevel.MEMORY_AND_DISK)
- D. itemsDf.store()
- E. itemsDf.cache()
正解: E
解説:
Explanation
The key to solving this question is knowing (or reading in the documentation) that, by default, cache() stores values to memory and writes any partitions for which there is insufficient memory to disk. persist() can achieve the exact same behavior, however not with the StorageLevel.MEMORY_ONLY option listed here. It is also worth noting that cache() does not have any arguments.
If you have troubles finding the storage level information in the documentation, please also see this student Q&A thread that sheds some light here.
Static notebook | Dynamic notebook: See test 2
質問 52
Which of the following code blocks displays the 10 rows with the smallest values of column value in DataFrame transactionsDf in a nicely formatted way?
- A. transactionsDf.sort(col("value").asc()).print(10)
- B. transactionsDf.sort(col("value")).show(10)
- C. transactionsDf.orderBy("value").asc().show(10)
- D. transactionsDf.sort(asc(value)).show(10)
- E. transactionsDf.sort(col("value").desc()).head()
正解: B
解説:
Explanation
show() is the correct method to look for here, since the question specifically asks for displaying the rows in a nicely formatted way. Here is the output of show (only a few rows shown):
+-------------+---------+-----+-------+---------+----+---------------+
|transactionId|predError|value|storeId|productId| f|transactionDate|
+-------------+---------+-----+-------+---------+----+---------------+
| 3| 3| 1| 25| 3|null| 1585824821|
| 5| null| 2| null| 2|null| 1575285427|
| 4| null| 3| 3| 2|null| 1583244275|
+-------------+---------+-----+-------+---------+----+---------------+
With regards to the sorting, specifically in ascending order since the smallest values should be shown first, the following expressions are valid:
- transactionsDf.sort(col("value")) ("ascending" is the default sort direction in the sort method)
- transactionsDf.sort(asc(col("value")))
- transactionsDf.sort(asc("value"))
- transactionsDf.sort(transactionsDf.value.asc())
- transactionsDf.sort(transactionsDf.value)
Also, orderBy is just an alias of sort, so all of these expressions work equally well using orderBy.
Static notebook | Dynamic notebook: See test 1
質問 53
Which of the following code blocks writes DataFrame itemsDf to disk at storage location filePath, making sure to substitute any existing data at that location?
- A. itemsDf.write.mode("overwrite").parquet(filePath)
- B. itemsDf.write.mode("overwrite").path(filePath)
- C. itemsDf.write(filePath, mode="overwrite")
- D. itemsDf.write.option("parquet").mode("overwrite").path(filePath)
- E. itemsDf.write().parquet(filePath, mode="overwrite")
正解: A
解説:
Explanation
itemsDf.write.mode("overwrite").parquet(filePath)
Correct! itemsDf.write returns a pyspark.sql.DataFrameWriter instance whose overwriting behavior can be modified via the mode setting or by passing mode="overwrite" to the parquet() command.
Although the parquet format is not prescribed for solving this question, parquet() is a valid operator to initiate Spark to write the data to disk.
itemsDf.write.mode("overwrite").path(filePath)
No. A pyspark.sql.DataFrameWriter instance does not have a path() method.
itemsDf.write.option("parquet").mode("overwrite").path(filePath)
Incorrect, see above. In addition, a file format cannot be passed via the option() method.
itemsDf.write(filePath, mode="overwrite")
Wrong. Unfortunately, this is too simple. You need to obtain access to a DataFrameWriter for the DataFrame through calling itemsDf.write upon which you can apply further methods to control how Spark data should be written to disk. You cannot, however, pass arguments to itemsDf.write directly.
itemsDf.write().parquet(filePath, mode="overwrite")
False. See above.
More info: pyspark.sql.DataFrameWriter.parquet - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
質問 54
The code block displayed below contains an error. The code block should arrange the rows of DataFrame transactionsDf using information from two columns in an ordered fashion, arranging first by column value, showing smaller numbers at the top and greater numbers at the bottom, and then by column predError, for which all values should be arranged in the inverse way of the order of items in column value. Find the error.
Code block:
transactionsDf.orderBy('value', asc_nulls_first(col('predError')))
- A. Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
- B. Column value should be wrapped by the col() operator.
- C. Column predError should be sorted in a descending way, putting nulls last.
- D. Column predError should be sorted by desc_nulls_first() instead.
- E. Instead of orderBy, sort should be used.
正解: C
解説:
Explanation
Correct code block:
transactionsDf.orderBy('value', desc_nulls_last('predError'))
Column predError should be sorted in a descending way, putting nulls last.
Correct! By default, Spark sorts ascending, putting nulls first. So, the inverse sort of the default sort is indeed desc_nulls_last.
Instead of orderBy, sort should be used.
No. DataFrame.sort() orders data per partition, it does not guarantee a global order. This is why orderBy is the more appropriate operator here.
Column value should be wrapped by the col() operator.
Incorrect. DataFrame.sort() accepts both string and Column objects.
Column predError should be sorted by desc_nulls_first() instead.
Wrong. Since Spark's default sort order matches asc_nulls_first(), nulls would have to come last when inverted.
Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
No, this would just sort the DataFrame by the very last column, but would not take information from both columns into account, as noted in the question.
More info: pyspark.sql.DataFrame.orderBy - PySpark 3.1.2 documentation, pyspark.sql.functions.desc_nulls_last - PySpark 3.1.2 documentation, sort() vs orderBy() in Spark | Towards Data Science Static notebook | Dynamic notebook: See test 3
質問 55
Which of the following code blocks returns a new DataFrame with the same columns as DataFrame transactionsDf, except for columns predError and value which should be removed?
- A. transactionsDf.drop(col("predError"), col("value"))
- B. transactionsDf.drop("predError", "value")
- C. transactionsDf.drop(predError, value)
- D. transactionsDf.drop(["predError", "value"])
- E. transactionsDf.drop("predError & value")
正解: B
解説:
Explanation
More info: pyspark.sql.DataFrame.drop - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
質問 56
The code block displayed below contains an error. The code block should return a DataFrame where all entries in column supplier contain the letter combination et in this order. Find the error.
Code block:
itemsDf.filter(Column('supplier').isin('et'))
- A. The expression inside the filter parenthesis is malformed and should be replaced by isin('et', 'supplier').
- B. The expression only returns a single column and filter should be replaced by select.
- C. The Column operator should be replaced by the col operator and instead of isin, contains should be used.
- D. Instead of isin, it should be checked whether column supplier contains the letters et, so isin should be replaced with contains. In addition, the column should be accessed using col['supplier'].
正解: A
解説:
Explanation
Correct code block:
itemsDf.filter(col('supplier').contains('et'))
A mixup can easily happen here between isin and contains. Since we want to check whether a column
"contains" the values et, this is the operator we should use here. Note that both methods are methods of Spark's Column object. See below for documentation links.
A specific Column object can be accessed through the col() method and not the Column() method or through col[], which is an essential thing to know here. In PySpark, Column references a generic column object. To use it for queries, you need to link the generic column object to a specific DataFrame. This can be achieved, for example, through the col() method.
More info:
- isin documentation: pyspark.sql.Column.isin - PySpark 3.1.1 documentation
- contains documentation: pyspark.sql.Column.contains - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 57
Which of the following code blocks returns only rows from DataFrame transactionsDf in which values in column productId are unique?
- A. transactionsDf.distinct("productId")
- B. transactionsDf.drop_duplicates(subset="productId")
- C. transactionsDf.dropDuplicates(subset="productId")
- D. transactionsDf.dropDuplicates(subset=["productId"])
- E. transactionsDf.unique("productId")
正解: D
解説:
Explanation
Although the question suggests using a method called unique() here, that method does not actually exist in PySpark. In PySpark, it is called distinct(). But then, this method is not the right one to use here, since with distinct() we could filter out unique values in a specific column.
However, we want to return the entire rows here. So the trick is to use dropDuplicates with the subset keyword parameter. In the documentation for dropDuplicates, the examples show that subset should be used with a list. And this is exactly the key to solving this question: The productId column needs to be fed into the subset argument in a list, even though it is just a single column.
More info: pyspark.sql.DataFrame.dropDuplicates - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 58
The code block shown below should return a copy of DataFrame transactionsDf with an added column cos.
This column should have the values in column value converted to degrees and having the cosine of those converted values taken, rounded to two decimals. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(__2__, round(__3__(__4__(__5__)),2))
- A. 1. withColumn
2. col("cos")
3. cos
4. degrees
5. col("value")
E
. 1. withColumn
2. "cos"
3. degrees
4. cos
5. col("value") - B. 1. withColumn
2. col("cos")
3. cos
4. degrees
5. transactionsDf.value - C. 1. withColumnRenamed
2. "cos"
3. cos
4. degrees
5. "transactionsDf.value" - D. 1. withColumn
2. "cos"
3. cos
4. degrees
5. transactionsDf.value
正解: D
解説:
Explanation
Correct code block:
transactionsDf.withColumn("cos", round(cos(degrees(transactionsDf.value)),2)) This question is especially confusing because col, "cos" are so similar. Similar-looking answer options can also appear in the exam and, just like in this question, you need to pay attention to the details to identify what the correct answer option is.
The first answer option to throw out is the one that starts with withColumnRenamed: The question NO:
speaks specifically of adding a column. The withColumnRenamed operator only renames an existing column, however, so you cannot use it here.
Next, you will have to decide what should be in gap 2, the first argument of transactionsDf.withColumn().
Looking at the documentation (linked below), you can find out that the first argument of withColumn actually needs to be a string with the name of the column to be added. So, any answer that includes col("cos") as the option for gap 2 can be disregarded.
This leaves you with two possible answers. The real difference between these two answers is where the cos and degree methods are, either in gaps 3 and 4, or vice-versa. From the question you can find out that the new column should have "the values in column value converted to degrees and having the cosine of those converted values taken". This prescribes you a clear order of operations: First, you convert values from column value to degrees and then you take the cosine of those values. So, the inner parenthesis (gap 4) should contain the degree method and then, logically, gap 3 holds the cos method. This leaves you with just one possible correct answer.
More info: pyspark.sql.DataFrame.withColumn - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
質問 59
The code block displayed below contains an error. The code block should configure Spark to split data in 20 parts when exchanging data between executors for joins or aggregations. Find the error.
Code block:
spark.conf.set(spark.sql.shuffle.partitions, 20)
- A. The code block sets the wrong option.
- B. The code block uses the wrong command for setting an option.
- C. The code block expresses the option incorrectly.
- D. The code block is missing a parameter.
- E. The code block sets the incorrect number of parts.
正解: C
解説:
Explanation
Correct code block:
spark.conf.set("spark.sql.shuffle.partitions", 20)
The code block expresses the option incorrectly.
Correct! The option should be expressed as a string.
The code block sets the wrong option.
No, spark.sql.shuffle.partitions is the correct option for the use case in the question.
The code block sets the incorrect number of parts.
Wrong, the code block correctly states 20 parts.
The code block uses the wrong command for setting an option.
No, in PySpark spark.conf.set() is the correct command for setting an option.
The code block is missing a parameter.
Incorrect, spark.conf.set() takes two parameters.
More info: Configuration - Spark 3.1.2 Documentation
質問 60
The code block displayed below contains an error. The code block should return all rows of DataFrame transactionsDf, but including only columns storeId and predError. Find the error.
Code block:
spark.collect(transactionsDf.select("storeId", "predError"))
- A. The take method should be used instead of the collect method.
- B. The collect method is not a method of the SparkSession object.
- C. Columns storeId and predError need to be represented as a Python list, so they need to be wrapped in brackets ([]).
- D. Instead of collect, collectAsRows needs to be called.
- E. Instead of select, DataFrame transactionsDf needs to be filtered using the filter operator.
正解: B
解説:
Explanation
Correct code block:
transactionsDf.select("storeId", "predError").collect()
collect() is a method of the DataFrame object.
More info: pyspark.sql.DataFrame.collect - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
質問 61
Which of the following code blocks returns a 2-column DataFrame that shows the distinct values in column productId and the number of rows with that productId in DataFrame transactionsDf?
- A. transactionsDf.count("productId").distinct()
- B. transactionsDf.groupBy("productId").agg(col("value").count())
- C. transactionsDf.count("productId")
- D. transactionsDf.groupBy("productId").count()
- E. transactionsDf.groupBy("productId").select(count("value"))
正解: D
解説:
Explanation
transactionsDf.groupBy("productId").count()
Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group.
transactionsDf.groupBy("productId").select(count("value"))
Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.
transactionsDf.count("productId")
No. DataFrame.count() does not take any arguments.
transactionsDf.count("productId").distinct()
Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right.
transactionsDf.groupBy("productId").agg(col("value").count())
False. A Column object, as returned by col("value"), does not have a count() method. You can see all available methods for Column object linked in the Spark documentation below.
More info: pyspark.sql.DataFrame.count - PySpark 3.1.2 documentation, pyspark.sql.Column - PySpark
3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 62
Which of the following statements about storage levels is incorrect?
- A. Caching can be undone using the DataFrame.unpersist() operator.
- B. DISK_ONLY will not use the worker node's memory.
- C. In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.
- D. MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.
- E. The cache operator on DataFrames is evaluated like a transformation.
正解: D
解説:
Explanation
MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.
Correct, this statement is wrong. Spark prioritizes storage in memory, and will only store data on disk that does not fit into memory.
DISK_ONLY will not use the worker node's memory.
Wrong, this statement is correct. DISK_ONLY keeps data only on the worker node's disk, but not in memory.
In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.
Wrong, this statement is correct. In fact, Spark does not have a provision to cache DataFrames in the driver (which sits on the edge node in client mode). Spark caches DataFrames in the executors' memory.
Caching can be undone using the DataFrame.unpersist() operator.
Wrong, this statement is correct. Caching, as achieved via the DataFrame.cache() or DataFrame.persist() operators can be undone using the DataFrame.unpersist() operator. This operator will remove all of its parts from the executors' memory and disk.
The cache operator on DataFrames is evaluated like a transformation.
Wrong, this statement is correct. DataFrame.cache() is evaluated like a transformation: Through lazy evaluation. This means that after calling DataFrame.cache() the command will not have any effect until you call a subsequent action, like DataFrame.cache().count().
More info: pyspark.sql.DataFrame.unpersist - PySpark 3.1.2 documentation
質問 63
Which of the following code blocks concatenates rows of DataFrames transactionsDf and transactionsNewDf, omitting any duplicates?
- A. transactionsDf.concat(transactionsNewDf).unique()
- B. transactionsDf.union(transactionsNewDf).unique()
- C. transactionsDf.join(transactionsNewDf, how="union").distinct()
- D. transactionsDf.union(transactionsNewDf).distinct()
- E. spark.union(transactionsDf, transactionsNewDf).distinct()
正解: D
解説:
Explanation
DataFrame.unique() and DataFrame.concat() do not exist and union() is not a method of the SparkSession. In addition, there is no union option for the join method in the DataFrame.join() statement.
More info: pyspark.sql.DataFrame.union - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
質問 64
Which of the following describes the difference between client and cluster execution modes?
- A. In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
- B. In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
- C. In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
- D. In cluster mode, the driver runs on the worker nodes, while the client mode runs the driver on the client machine.
- E. In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.
正解: D
解説:
Explanation
In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.
This is wrong, since execution modes do not specify whether workloads are run in the cloud or on-premise.
In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
Wrong, since in both cases executors run on worker nodes.
In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
Wrong - in cluster mode, the driver runs on a worker node. In client mode, the driver runs on the client machine.
In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
No. In both modes, the cluster manager is typically on a separate node - not on the same host as the driver. It only runs on the same host as the driver in local execution mode.
More info: Learning Spark, 2nd Edition, Chapter 1, and Spark: The Definitive Guide, Chapter 15. ()
質問 65
The code block displayed below contains an error. The code block below is intended to add a column itemNameElements to DataFrame itemsDf that includes an array of all words in column itemName. Find the error.
Sample of DataFrame itemsDf:
1.+------+----------------------------------+-------------------+
2.|itemId|itemName |supplier |
3.+------+----------------------------------+-------------------+
4.|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|
5.|2 |Elegant Outdoors Summer Dress |YetiX |
6.|3 |Outdoors Backpack |Sports Company Inc.|
7.+------+----------------------------------+-------------------+
Code block:
itemsDf.withColumnRenamed("itemNameElements", split("itemName"))
itemsDf.withColumnRenamed("itemNameElements", split("itemName"))
- A. Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument "
" needs to be passed to the split method. - B. The expressions "itemNameElements" and split("itemName") need to be swapped.
- C. Operator withColumnRenamed needs to be replaced with operator withColumn and the split method needs to be replaced by the splitString method.
- D. All column names need to be wrapped in the col() operator.
- E. Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument
"," needs to be passed to the split method.
正解: A
解説:
Explanation
Correct code block:
itemsDf.withColumn("itemNameElements", split("itemName"," "))
Output of code block:
+------+----------------------------------+-------------------+------------------------------------------+
|itemId|itemName |supplier |itemNameElements |
+------+----------------------------------+-------------------+------------------------------------------+
|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|[Thick, Coat, for, Walking, in, the, Snow]|
|2 |Elegant Outdoors Summer Dress |YetiX |[Elegant, Outdoors, Summer, Dress] |
|3 |Outdoors Backpack |Sports Company Inc.|[Outdoors, Backpack] |
+------+----------------------------------+-------------------+------------------------------------------+ The key to solving this question is that the split method definitely needs a second argument here (also look at the link to the documentation below). Given the values in column itemName in DataFrame itemsDf, this should be a space character " ". This is the character we need to split the words in the column.
More info: pyspark.sql.functions.split - PySpark 3.1.1 documentation
Static notebook | Dynamic notebook: See test 1
質問 66
Which of the following code blocks silently writes DataFrame itemsDf in avro format to location fileLocation if a file does not yet exist at that location?
- A. itemsDf.save.format("avro").mode("ignore").write(fileLocation)
- B. itemsDf.write.format("avro").mode("errorifexists").save(fileLocation)
- C. itemsDf.write.format("avro").mode("ignore").save(fileLocation)
- D. itemsDf.write.avro(fileLocation)
- E. spark.DataFrameWriter(itemsDf).format("avro").write(fileLocation)
正解: D
解説:
Explanation
The trick in this question is knowing the "modes" of the DataFrameWriter. Mode ignore will ignore if a file already exists and not replace that file, but also not throw an error. Mode errorifexists will throw an error, and is the default mode of the DataFrameWriter. The question NO:
explicitly calls for the DataFrame to be "silently" written if it does not exist, so you need to specify mode("ignore") here to avoid having Spark communicate any error to you if the file already exists.
The `overwrite' mode would not be right here, since, although it would be silent, it would overwrite the already-existing file. This is not what the question asks for.
It is worth noting that the option starting with spark.DataFrameWriter(itemsDf) cannot work, since spark references the SparkSession object, but that object does not provide the DataFrameWriter.
As you can see in the documentation (below), DataFrameWriter is part of PySpark's SQL API, but not of its SparkSession API.
More info:
DataFrameWriter: pyspark.sql.DataFrameWriter.save - PySpark 3.1.1 documentation SparkSession API: Spark SQL - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 67
Which of the following describes Spark's standalone deployment mode?
- A. Standalone mode uses a single JVM to run Spark driver and executor processes.
- B. Standalone mode means that the cluster does not contain the driver.
- C. Standalone mode uses only a single executor per worker per application.
- D. Standalone mode is a viable solution for clusters that run multiple frameworks, not only Spark.
- E. Standalone mode is how Spark runs on YARN and Mesos clusters.
正解: C
解説:
Explanation
Standalone mode uses only a single executor per worker per application.
This is correct and a limitation of Spark's standalone mode.
Standalone mode is a viable solution for clusters that run multiple frameworks.
Incorrect. A limitation of standalone mode is that Apache Spark must be the only framework running on the cluster. If you would want to run multiple frameworks on the same cluster in parallel, for example Apache Spark and Apache Flink, you would consider the YARN deployment mode.
Standalone mode uses a single JVM to run Spark driver and executor processes.
No, this is what local mode does.
Standalone mode is how Spark runs on YARN and Mesos clusters.
No. YARN and Mesos modes are two deployment modes that are different from standalone mode. These modes allow Spark to run alongside other frameworks on a cluster. When Spark is run in standalone mode, only the Spark framework can run on the cluster.
Standalone mode means that the cluster does not contain the driver.
Incorrect, the cluster does not contain the driver in client mode, but in standalone mode the driver runs on a node in the cluster.
More info: Learning Spark, 2nd Edition, Chapter 1
質問 68
Which of the following code blocks shows the structure of a DataFrame in a tree-like way, containing both column names and types?
- A. itemsDf.print.schema()
- B. spark.schema(itemsDf)
- C. 1.print(itemsDf.columns)
2.print(itemsDf.types) - D. itemsDf.rdd.printSchema()
- E. itemsDf.printSchema()
正解: E
解説:
Explanation
itemsDf.printSchema()
Correct! Here is an example of what itemsDf.printSchema() shows, you can see the tree-like structure containing both column names and types:
root
|-- itemId: integer (nullable = true)
|-- attributes: array (nullable = true)
| |-- element: string (containsNull = true)
|-- supplier: string (nullable = true)
itemsDf.rdd.printSchema()
No, the DataFrame's underlying RDD does not have a printSchema() method.
spark.schema(itemsDf)
Incorrect, there is no spark.schema command.
print(itemsDf.columns)
print(itemsDf.dtypes)
Wrong. While the output of this code blocks contains both column names and column types, the information is not arranges in a tree-like way.
itemsDf.print.schema()
No, DataFrame does not have a print method.
Static notebook | Dynamic notebook: See test 3
質問 69
Which of the following statements about lazy evaluation is incorrect?
- A. Predicate pushdown is a feature resulting from lazy evaluation.
- B. Spark will fail a job only during execution, but not during definition.
- C. Lineages allow Spark to coalesce transformations into stages
- D. Accumulators do not change the lazy evaluation model of Spark.
- E. Execution is triggered by transformations.
正解: E
解説:
Explanation
Execution is triggered by transformations.
Correct. Execution is triggered by actions only, not by transformations.
Lineages allow Spark to coalesce transformations into stages.
Incorrect. In Spark, lineage means a recording of transformations. This lineage enables lazy evaluation in Spark.
Predicate pushdown is a feature resulting from lazy evaluation.
Wrong. Predicate pushdown means that, for example, Spark will execute filters as early in the process as possible so that it deals with the least possible amount of data in subsequent transformations, resulting in a performance improvements.
Accumulators do not change the lazy evaluation model of Spark.
Incorrect. In Spark, accumulators are only updated when the query that refers to the is actually executed. In other words, they are not updated if the query is not (yet) executed due to lazy evaluation.
Spark will fail a job only during execution, but not during definition.
Wrong. During definition, due to lazy evaluation, the job is not executed and thus certain errors, for example reading from a non-existing file, cannot be caught. To be caught, the job needs to be executed, for example through an action.
質問 70
Which of the following describes a narrow transformation?
- A. narrow transformation is an operation in which data is exchanged across partitions.
- B. A narrow transformation is a process in which data from multiple RDDs is used.
- C. A narrow transformation is an operation in which no data is exchanged across the cluster.
- D. A narrow transformation is an operation in which data is exchanged across the cluster.
- E. A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like 16-bit or 8-bit float variables.
正解: C
解説:
Explanation
A narrow transformation is an operation in which no data is exchanged across the cluster.
Correct! In narrow transformations, no data is exchanged across the cluster, since these transformations do not require any data from outside of the partition they are applied on. Typical narrow transformations include filter, drop, and coalesce.
A narrow transformation is an operation in which data is exchanged across partitions.
No, that would be one definition of a wide transformation, but not of a narrow transformation. Wide transformations typically cause a shuffle, in which data is exchanged across partitions, executors, and the cluster.
A narrow transformation is an operation in which data is exchanged across the cluster.
No, see explanation just above this one.
A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like
16-bit or 8-bit float variables.
No, type conversion has nothing to do with narrow transformations in Spark.
A narrow transformation is a process in which data from multiple RDDs is used.
No. A resilient distributed dataset (RDD) can be described as a collection of partitions. In a narrow transformation, no data is exchanged between partitions. Thus, no data is exchanged between RDDs.
One could say though that a narrow transformation and, in fact, any transformation results in a new RDD being created. This is because a transformation results in a change to an existing RDD (RDDs are the foundation of other Spark data structures, like DataFrames). But, since RDDs are immutable, a new RDD needs to be created to reflect the change caused by the transformation.
More info: Spark Transformation and Action: A Deep Dive | by Misbah Uddin | CodeX | Medium
質問 71
Which of the following describes how Spark achieves fault tolerance?
- A. Spark builds a fault-tolerant layer on top of the legacy RDD data system, which by itself is not fault tolerant.
- B. Spark helps fast recovery of data in case of a worker fault by providing the MEMORY_AND_DISK storage level option.
- C. Spark is only fault-tolerant if this feature is specifically enabled via the spark.fault_recovery.enabled property.
- D. Due to the mutability of DataFrames after transformations, Spark reproduces them using observed lineage in case of worker node failure.
- E. If an executor on a worker node fails while calculating an RDD, that RDD can be recomputed by another executor using the lineage.
正解: E
解説:
Explanation
Due to the mutability of DataFrames after transformations, Spark reproduces them using observed lineage in case of worker node failure.
Wrong - Between transformations, DataFrames are immutable. Given that Spark also records the lineage, Spark can reproduce any DataFrame in case of failure. These two aspects are the key to understanding fault tolerance in Spark.
Spark builds a fault-tolerant layer on top of the legacy RDD data system, which by itself is not fault tolerant.
Wrong. RDD stands for Resilient Distributed Dataset and it is at the core of Spark and not a "legacy system".
It is fault-tolerant by design.
Spark helps fast recovery of data in case of a worker fault by providing the MEMORY_AND_DISK storage level option.
This is not true. For supporting recovery in case of worker failures, Spark provides "_2", "_3", and so on, storage level options, for example MEMORY_AND_DISK_2. These storage levels are specifically designed to keep duplicates of the data on multiple nodes. This saves time in case of a worker fault, since a copy of the data can be used immediately, vs. having to recompute it first.
Spark is only fault-tolerant if this feature is specifically enabled via the spark.fault_recovery.enabled property.
No, Spark is fault-tolerant by design.
質問 72
......
Associate-Developer-Apache-Spark別格な問題集で最上級の成績にさせるAssociate-Developer-Apache-Spark問題：https://jp.fast2test.com/Associate-Developer-Apache-Spark-premium-file.html
手に入れよう！最新Associate-Developer-Apache-Spark認定の有効な試験問題集解答：https://drive.google.com/open?id=1iXTRIAawpQAStgIN7QLWpHI0IyLvHG7q