有効なDatabricks-Machine-Learning-Associateテスト解答Databricks Databricks-Machine-Learning-Associate試験PDF問題を試そう
Databricks Databricks-Machine-Learning-Associate認定リアル2025年最新の模擬試験合格させます
Databricks Databricks-Machine-Learning-Associate 認定試験の出題範囲：
| トピック | 出題範囲 |
|---|---|
| トピック 1 |
|
| トピック 2 |
|
| トピック 3 |
|
| トピック 4 |
|
質問 # 23
The implementation of linear regression in Spark ML first attempts to solve the linear regression problem using matrix decomposition, but this method does not scale well to large datasets with a large number of variables.
Which of the following approaches does Spark ML use to distribute the training of a linear regression model for large data?
- A. Least-squares method
- B. Iterative optimization
- C. Logistic regression
- D. Singular value decomposition
- E. Spark ML cannot distribute linear regression training
正解：B
解説：
For large datasets with many variables, Spark ML distributes the training of a linear regression model using iterative optimization methods. Specifically, Spark ML employs algorithms such as Gradient Descent or L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) to iteratively minimize the loss function. These iterative methods are suitable for distributed computing environments and can handle large-scale data efficiently by partitioning the data across nodes in a cluster and performing parallel updates.
Reference:
Spark MLlib Documentation (Linear Regression with Iterative Optimization).
質問 # 24
A data scientist is using Spark ML to engineer features for an exploratory machine learning project.
They decide they want to standardize their features using the following code block:
Upon code review, a colleague expressed concern with the features being standardized prior to splitting the data into a training set and a test set.
Which of the following changes can the data scientist make to address the concern?
- A. Utilize the MinMaxScaler object to standardize the test data according to global minimum and maximum values
- B. Utilize the Pipeline API to standardize the test data according to the training data's summary statistics
- C. Utilize the Pipeline API to standardize the training data according to the test data's summary statistics
- D. Utilize a cross-validation process rather than a train-test split process to remove the need for standardizing data
- E. Utilize the MinMaxScaler object to standardize the training data according to global minimum and maximum values
正解：B
解説：
To address the concern about standardizing features prior to splitting the data, the correct approach is to use the Pipeline API to ensure that only the training data's summary statistics are used to standardize the test data. This is achieved by fitting the StandardScaler (or any scaler) on the training data and then transforming both the training and test data using the fitted scaler. This approach prevents information leakage from the test data into the model training process and ensures that the model is evaluated fairly.
Reference:
Best Practices in Preprocessing in Spark ML (Handling Data Splits and Feature Standardization).
質問 # 25
A data scientist wants to efficiently tune the hyperparameters of a scikit-learn model in parallel. They elect to use the Hyperopt library to facilitate this process.
Which of the following Hyperopt tools provides the ability to optimize hyperparameters in parallel?
- A. SparkTrials
- B. fmin
- C. search_space
- D. objective_function
- E. quniform
正解：A
解説：
The SparkTrials class in the Hyperopt library allows for parallel hyperparameter optimization on a Spark cluster. This enables efficient tuning of hyperparameters by distributing the optimization process across multiple nodes in a cluster.
from hyperopt import fmin, tpe, hp, SparkTrials search_space = { 'x': hp.uniform('x', 0, 1), 'y': hp.uniform('y', 0, 1) } def objective(params): return params['x'] ** 2 + params['y'] ** 2 spark_trials = SparkTrials(parallelism=4) best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=100, trials=spark_trials) Reference:
Hyperopt Documentation
質問 # 26
A data scientist has written a feature engineering notebook that utilizes the pandas library. As the size of the data processed by the notebook increases, the notebook's runtime is drastically increasing, but it is processing slowly as the size of the data included in the process increases.
Which of the following tools can the data scientist use to spend the least amount of time refactoring their notebook to scale with big data?
- A. Feature Store
- B. pandas API on Spark
- C. Spark SQL
- D. PySpark DataFrame API
正解：B
解説：
The pandas API on Spark provides a way to scale pandas operations to big data while minimizing the need for refactoring existing pandas code. It allows users to run pandas operations on Spark DataFrames, leveraging Spark's distributed computing capabilities to handle large datasets more efficiently. This approach requires minimal changes to the existing code, making it a convenient option for scaling pandas-based feature engineering notebooks.
Reference:
Databricks documentation on pandas API on Spark: pandas API on Spark
質問 # 27
A new data scientist has started working on an existing machine learning project. The project is a scheduled Job that retrains every day. The project currently exists in a Repo in Databricks. The data scientist has been tasked with improving the feature engineering of the pipeline's preprocessing stage. The data scientist wants to make necessary updates to the code that can be easily adopted into the project without changing what is being run each day.
Which approach should the data scientist take to complete this task?
- A. They can clone the notebooks in the repository into a Databricks Workspace folder and make the necessary changes.
- B. They can create a new Git repository, import it into Databricks, and copy and paste the existing code from the original repository before making changes.
- C. They can clone the notebooks in the repository into a new Databricks Repo and make the necessary changes.
- D. They can create a new branch in Databricks, commit their changes, and push those changes to the Git provider.
正解：D
解説：
The best approach for the data scientist to take in this scenario is to create a new branch in Databricks, commit their changes, and push those changes to the Git provider. This approach allows the data scientist to make updates and improvements to the feature engineering part of the preprocessing pipeline without affecting the main codebase that runs daily. By creating a new branch, they can work on their changes in isolation. Once the changes are ready and tested, they can be merged back into the main branch through a pull request, ensuring a smooth integration process and allowing for code review and collaboration with other team members.
Reference:
Databricks documentation on Git integration: Databricks Repos
質問 # 28
Which of the following tools can be used to distribute large-scale feature engineering without the use of a UDF or pandas Function API for machine learning pipelines?
- A. Keras
- B. PyTorch
- C. Scikit-learn
- D. Spark ML
正解：D
解説：
Spark MLlib is a machine learning library within Apache Spark that provides scalable and distributed machine learning algorithms. It is designed to work with Spark DataFrames and leverages Spark's distributed computing capabilities to perform large-scale feature engineering and model training without the need for user-defined functions (UDFs) or the pandas Function API. Spark MLlib provides built-in transformations and algorithms that can be applied directly to large datasets.
Reference:
Databricks documentation on Spark MLlib: Spark MLlib
質問 # 29
A data scientist wants to parallelize the training of trees in a gradient boosted tree to speed up the training process. A colleague suggests that parallelizing a boosted tree algorithm can be difficult.
Which of the following describes why?
- A. Gradient boosting calculates gradients in evaluation metrics using all cores which prevents parallelization.
- B. Gradient boosting is an iterative algorithm that requires information from the previous iteration to perform the next step.
- C. Gradient boosting requires access to all data at once which cannot happen during parallelization.
- D. Gradient boosting is not a linear algebra-based algorithm which is required for parallelization
正解：B
解説：
Gradient boosting is fundamentally an iterative algorithm where each new tree is built based on the errors of the previous ones. This sequential dependency makes it difficult to parallelize the training of trees in gradient boosting, as each step relies on the results from the preceding step. Parallelization in this context would undermine the core methodology of the algorithm, which depends on sequentially improving the model's performance with each iteration.
Reference:
Machine Learning Algorithms (Challenges with Parallelizing Gradient Boosting).
Gradient boosting is an ensemble learning technique that builds models in a sequential manner. Each new model corrects the errors made by the previous ones. This sequential dependency means that each iteration requires the results of the previous iteration to make corrections. Here is a step-by-step explanation of why this makes parallelization challenging:
Sequential Nature: Gradient boosting builds one tree at a time. Each tree is trained to correct the residual errors of the previous trees. This requires the model to complete one iteration before starting the next.
Dependence on Previous Iterations: The gradient calculation at each step depends on the predictions made by the previous models. Therefore, the model must wait until the previous tree has been fully trained and evaluated before starting to train the next tree.
Difficulty in Parallelization: Because of this dependency, it is challenging to parallelize the training process. Unlike algorithms that process data independently in each step (e.g., random forests), gradient boosting cannot easily distribute the work across multiple processors or cores for simultaneous execution.
This iterative and dependent nature of the gradient boosting process makes it difficult to parallelize effectively.
Reference
Gradient Boosting Machine Learning Algorithm
Understanding Gradient Boosting Machines
質問 # 30
A data scientist is performing hyperparameter tuning using an iterative optimization algorithm. Each evaluation of unique hyperparameter values is being trained on a single compute node. They are performing eight total evaluations across eight total compute nodes. While the accuracy of the model does vary over the eight evaluations, they notice there is no trend of improvement in the accuracy. The data scientist believes this is due to the parallelization of the tuning process.
Which change could the data scientist make to improve their model accuracy over the course of their tuning process?
- A. Change the number of compute nodes and the number of evaluations to be much larger but equal.
- B. Change the number of compute nodes to be double or more than double the number of evaluations.
- C. Change the iterative optimization algorithm used to facilitate the tuning process.
- D. Change the number of compute nodes to be half or less than half of the number of evaluations.
正解：C
解説：
The lack of improvement in model accuracy across evaluations suggests that the optimization algorithm might not be effectively exploring the hyperparameter space. Iterative optimization algorithms like Tree-structured Parzen Estimators (TPE) or Bayesian Optimization can adapt based on previous evaluations, guiding the search towards more promising regions of the hyperparameter space.
Changing the optimization algorithm can lead to better utilization of the information gathered during each evaluation, potentially improving the overall accuracy.
Reference:
Hyperparameter Optimization with Hyperopt
質問 # 31
A health organization is developing a classification model to determine whether or not a patient currently has a specific type of infection. The organization's leaders want to maximize the number of positive cases identified by the model.
Which of the following classification metrics should be used to evaluate the model?
- A. Precision
- B. Recall
- C. Area under the residual operating curve
- D. RMSE
- E. Accuracy
正解：B
解説：
When the goal is to maximize the identification of positive cases in a classification task, the metric of interest is Recall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e., the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of performance measurement and are not specifically focused on maximizing the detection of positive cases alone.
Reference:
Classification Metrics in Machine Learning (Understanding Recall).
質問 # 32
A data scientist is wanting to explore summary statistics for Spark DataFrame spark_df. The data scientist wants to see the count, mean, standard deviation, minimum, maximum, and interquartile range (IQR) for each numerical feature.
Which of the following lines of code can the data scientist run to accomplish the task?
- A. spark_df.printSchema()
- B. spark_df.toPandas()
- C. spark_df.stats()
- D. spark_df.describe().head()
- E. spark_df.summary ()
正解：E
解説：
The summary() function in PySpark's DataFrame API provides descriptive statistics which include count, mean, standard deviation, min, max, and quantiles for numeric columns. Here are the steps on how it can be used:
Import PySpark: Ensure PySpark is installed and correctly configured in the Databricks environment.
Load Data: Load the data into a Spark DataFrame.
Apply Summary: Use spark_df.summary() to generate summary statistics.
View Results: The output from the summary() function includes the statistics specified in the query (count, mean, standard deviation, min, max, and potentially quartiles which approximate the interquartile range).
Reference
PySpark Documentation: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.summary.html
質問 # 33
A data scientist is utilizing MLflow Autologging to automatically track their machine learning experiments. After completing a series of runs for the experiment experiment_id, the data scientist wants to identify the run_id of the run with the best root-mean-square error (RMSE).
Which of the following lines of code can be used to identify the run_id of the run with the best RMSE in experiment_id?
- A.
- B.
- C.
- D.
正解：D
解説：
To find the run_id of the run with the best root-mean-square error (RMSE) in an MLflow experiment, the correct line of code to use is:
mlflow.search_runs( experiment_id, order_by=["metrics.rmse"] )["run_id"][0] This line of code searches the runs in the specified experiment, orders them by the RMSE metric in ascending order (the lower the RMSE, the better), and retrieves the run_id of the best-performing run. Option C correctly represents this logic.
Reference
MLflow documentation on tracking experiments: https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.search_runs
質問 # 34
A machine learning engineer is trying to scale a machine learning pipeline by distributing its feature engineering process.
Which of the following feature engineering tasks will be the least efficient to distribute?
- A. Creating binary indicator features for missing values
- B. Imputing missing feature values with the true median
- C. One-hot encoding categorical features
- D. Target encoding categorical features
- E. Imputing missing feature values with the mean
正解：B
解説：
Among the options listed, calculating the true median for imputing missing feature values is the least efficient to distribute. This is because the true median requires knowledge of the entire data distribution, which can be computationally expensive in a distributed environment. Unlike mean or mode, finding the median requires sorting the data or maintaining a full distribution, which is more intensive and often requires shuffling the data across partitions.
Reference
Challenges in parallel processing and distributed computing for data aggregation like median calculation: https://www.apache.org
質問 # 35
A data scientist has defined a Pandas UDF function predict to parallelize the inference process for a single-node model:
They have written the following incomplete code block to use predict to score each record of Spark DataFrame spark_df:
Which of the following lines of code can be used to complete the code block to successfully complete the task?
- A. mapInPandas(predict)
- B. mapInPandas(predict(spark_df.columns))
- C. predict(spark_df.columns)
- D. predict(Iterator(spark_df))
- E. predict(*spark_df.columns)
正解：A
解説：
To apply the Pandas UDF predict to each record of a Spark DataFrame, you use the mapInPandas method. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predict in this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments or incorrect function calls.
Reference:
PySpark DataFrame documentation (Using mapInPandas with UDFs).
質問 # 36
A data scientist has created two linear regression models. The first model uses price as a label variable and the second model uses log(price) as a label variable. When evaluating the RMSE of each model by comparing the label predictions to the actual price values, the data scientist notices that the RMSE for the second model is much larger than the RMSE of the first model.
Which of the following possible explanations for this difference is invalid?
- A. The data scientist failed to exponentiate the predictions in the second model prior to computing the RMSE
- B. The first model is much more accurate than the second model
- C. The second model is much more accurate than the first model
- D. The data scientist failed to take the log of the predictions in the first model prior to computing the RMSE
- E. The RMSE is an invalid evaluation metric for regression problems
正解：E
解説：
The Root Mean Squared Error (RMSE) is a standard and widely used metric for evaluating the accuracy of regression models. The statement that it is invalid is incorrect. Here's a breakdown of why the other statements are or are not valid:
Transformations and RMSE Calculation: If the model predictions were transformed (e.g., using log), they should be converted back to their original scale before calculating RMSE to ensure accuracy in the evaluation. Missteps in this conversion process can lead to misleading RMSE values.
Accuracy of Models: Without additional information, we can't definitively say which model is more accurate without considering their RMSE values properly scaled back to the original price scale.
Appropriateness of RMSE: RMSE is entirely valid for regression problems as it provides a measure of how accurately a model predicts the outcome, expressed in the same units as the dependent variable.
Reference
"Applied Predictive Modeling" by Max Kuhn and Kjell Johnson (Springer, 2013), particularly the chapters discussing model evaluation metrics.
質問 # 37
Which of the following is a benefit of using vectorized pandas UDFs instead of standard PySpark UDFs?
- A. The vectorized pandas UDFs work on distributed DataFrames
- B. The vectorized pandas UDFs allow for the use of type hints
- C. The vectorized pandas UDFs process data in memory rather than spilling to disk
- D. The vectorized pandas UDFs allow for pandas API use inside of the function
- E. The vectorized pandas UDFs process data in batches rather than one row at a time
正解：E
解説：
Vectorized pandas UDFs, also known as Pandas UDFs, are a powerful feature in PySpark that allows for more efficient operations than standard UDFs. They operate by processing data in batches, utilizing vectorized operations that leverage pandas to perform operations on whole batches of data at once. This approach is much more efficient than processing data row by row as is typical with standard PySpark UDFs, which can significantly speed up the computation.
Reference
PySpark Documentation on UDFs: https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html#pandas-udfs-a-k-a-vectorized-udfs
質問 # 38
A data scientist learned during their training to always use 5-fold cross-validation in their model development workflow. A colleague suggests that there are cases where a train-validation split could be preferred over k-fold cross-validation when k > 2.
Which of the following describes a potential benefit of using a train-validation split over k-fold cross-validation in this scenario?
- A. Reproducibility is achievable when using a train-validation split
- B. A holdout set is not necessary when using a train-validation split
- C. Fewer models need to be trained when using a train-validation split
- D. Bias is avoidable when using a train-validation split
- E. Fewer hyperparameter values need to be tested when using a train-validation split
正解：C
質問 # 39
A data scientist has been given an incomplete notebook from the data engineering team. The notebook uses a Spark DataFrame spark_df on which the data scientist needs to perform further feature engineering. Unfortunately, the data scientist has not yet learned the PySpark DataFrame API.
Which of the following blocks of code can the data scientist run to be able to use the pandas API on Spark?
- A. import pandas as pd
df = pd.DataFrame(spark_df) - B. import pyspark.pandas as ps
df = ps.to_pandas(spark_df) - C. import pyspark.pandas as ps
df = ps.DataFrame(spark_df) - D. spark_df.to_pandas()
正解：C
解説：
To use the pandas API on Spark, the data scientist can run the following code block:
import pyspark.pandas as ps df = ps.DataFrame(spark_df)
This code imports the pandas API on Spark and converts the Spark DataFrame spark_df into a pandas-on-Spark DataFrame, allowing the data scientist to use familiar pandas functions for further feature engineering.
Reference:
Databricks documentation on pandas API on Spark: pandas API on Spark
質問 # 40
Which of the following describes the relationship between native Spark DataFrames and pandas API on Spark DataFrames?
- A. pandas API on Spark DataFrames are less mutable versions of Spark DataFrames
- B. pandas API on Spark DataFrames are more performant than Spark DataFrames
- C. pandas API on Spark DataFrames are single-node versions of Spark DataFrames with additional metadata
- D. pandas API on Spark DataFrames are made up of Spark DataFrames and additional metadata
正解：D
解説：
The pandas API on Spark DataFrames are made up of Spark DataFrames with additional metadata. The pandas API on Spark aims to provide the pandas-like experience with the scalability and distributed nature of Spark. It allows users to work with pandas functions on large datasets by leveraging Spark's underlying capabilities.
Reference:
Databricks documentation on pandas API on Spark: pandas API on Spark
質問 # 41
A data scientist has produced three new models for a single machine learning problem. In the past, the solution used just one model. All four models have nearly the same prediction latency, but a machine learning engineer suggests that the new solution will be less time efficient during inference.
In which situation will the machine learning engineer be correct?
- A. When the new solution requires that each model computes a prediction for every record
- B. When the new solution's models have an average latency that is larger than the size of the original model
- C. When the new solution requires the use of fewer feature variables than the original model
- D. When the new solution's models have an average size that is larger than the size of the original model
- E. When the new solution requires if-else logic determining which model to use to compute each prediction
正解：A
解説：
If the new solution requires that each of the three models computes a prediction for every record, the time efficiency during inference will be reduced. This is because the inference process now involves running multiple models instead of a single model, thereby increasing the overall computation time for each record.
In scenarios where inference must be done by multiple models for each record, the latency accumulates, making the process less time efficient compared to using a single model.
Reference:
Model Ensemble Techniques
質問 # 42
A data scientist uses 3-fold cross-validation when optimizing model hyperparameters for a regression problem. The following root-mean-squared-error values are calculated on each of the validation folds:
* 10.0
* 12.0
* 17.0
Which of the following values represents the overall cross-validation root-mean-squared error?
- A. 10.0
- B. 17.0
- C. 12.0
- D. 39.0
- E. 13.0
正解：E
解説：
To calculate the overall cross-validation root-mean-squared error (RMSE), you average the RMSE values obtained from each validation fold. Given the RMSE values of 10.0, 12.0, and 17.0 for the three folds, the overall cross-validation RMSE is calculated as the average of these three values:
Overall CV RMSE=10.0+12.0+17.03=39.03=13.0Overall CV RMSE=310.0+12.0+17.0=339.0=13.0 Thus, the correct answer is 13.0, which accurately represents the average RMSE across all folds.
Reference:
Cross-validation in Regression (Understanding Cross-Validation Metrics).
質問 # 43
A data scientist has been given an incomplete notebook from the data engineering team. The notebook uses a Spark DataFrame spark_df on which the data scientist needs to perform further feature engineering. Unfortunately, the data scientist has not yet learned the PySpark DataFrame API.
Which of the following blocks of code can the data scientist run to be able to use the pandas API on Spark?
- A. spark_df.to_sql()
- B. import pandas as pd
df = pd.DataFrame(spark_df) - C. import pyspark.pandas as ps
df = ps.to_pandas(spark_df) - D. import pyspark.pandas as ps
df = ps.DataFrame(spark_df) - E. spark_df.to_pandas()
正解：D
解説：
To use the pandas API on Spark, which is designed to bridge the gap between the simplicity of pandas and the scalability of Spark, the correct approach involves importing the pyspark.pandas (recently renamed to pandas_api_on_spark) module and converting a Spark DataFrame to a pandas-on-Spark DataFrame using this API. The provided syntax correctly initializes a pandas-on-Spark DataFrame, allowing the data scientist to work with the familiar pandas-like API on large datasets managed by Spark.
Reference
Pandas API on Spark Documentation: https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html
質問 # 44
What is the name of the method that transforms categorical features into a series of binary indicator feature variables?
- A. Leave-one-out encoding
- B. Target encoding
- C. String indexing
- D. Categorical
- E. One-hot encoding
正解：E
解説：
The method that transforms categorical features into a series of binary indicator variables is known as one-hot encoding. This technique converts each categorical value into a new binary column, which is essential for models that require numerical input. One-hot encoding is widely used because it helps to handle categorical data without introducing a false ordinal relationship among categories.
Reference:
Feature Engineering Techniques (One-Hot Encoding).
質問 # 45
......
Databricks-Machine-Learning-Associate試験問題と有効なDatabricks-Machine-Learning-Associate問題集PDF：https://jp.fast2test.com/Databricks-Machine-Learning-Associate-premium-file.html