[2025年02月]更新のDatabricks Databricks-Machine-Learning-Associate公式認定ガイドPDF
試験Databricks-Machine-Learning-Associate Databricks Certified Machine Learning Associate Exam
Databricks Databricks-Machine-Learning-Associate 認定試験の出題範囲：
| トピック | 出題範囲 |
|---|---|
| トピック 1 |
|
| トピック 2 |
|
| トピック 3 |
|
| トピック 4 |
|
質問 # 24
A machine learning engineer is trying to scale a machine learning pipeline pipeline that contains multiple feature engineering stages and a modeling stage. As part of the cross-validation process, they are using the following code block:
A colleague suggests that the code block can be changed to speed up the tuning process by passing the model object to the estimator parameter and then placing the updated cv object as the final stage of the pipeline in place of the original model.
Which of the following is a negative consequence of the approach suggested by the colleague?
- A. The cross-validation process will no longer be reproducible
- B. The feature engineering stages will be computed using validation data
- C. The model will take longer to train for each unique combination of hvperparameter values
- D. The cross-validation process will no longer be
- E. The model will be refit one more per cross-validation fold
正解：B
解説：
If the model object is passed to the estimator parameter of CrossValidator and the cross-validation object itself is placed as a stage in the pipeline, the feature engineering stages within the pipeline would be applied separately to each training and validation fold during cross-validation. This leads to a significant issue: the feature engineering stages would be computed using validation data, thereby leaking information from the validation set into the training process. This would potentially invalidate the cross-validation results by giving an overly optimistic performance estimate.
Reference:
Cross-validation and Pipeline Integration in MLlib (Avoiding Data Leakage in Pipelines).
質問 # 25
A data scientist has a Spark DataFrame spark_df. They want to create a new Spark DataFrame that contains only the rows from spark_df where the value in column discount is less than or equal 0.
Which of the following code blocks will accomplish this task?
- A. spark_df[spark_df["discount"] <= 0]
- B. spark_df.filter (col("discount") <= 0)
- C. spark_df.loc(spark_df["discount"] <= 0, :]
- D. spark_df.loc[:,spark_df["discount"] <= 0]
正解：B
解説：
To filter rows in a Spark DataFrame based on a condition, the filter method is used. In this case, the condition is that the value in the "discount" column should be less than or equal to 0. The correct syntax uses the filter method along with the col function from pyspark.sql.functions.
Correct code:
from pyspark.sql.functions import col filtered_df = spark_df.filter(col("discount") <= 0) Option A and D use Pandas syntax, which is not applicable in PySpark. Option B is closer but misses the use of the col function.
Reference:
PySpark SQL Documentation
質問 # 26
A new data scientist has started working on an existing machine learning project. The project is a scheduled Job that retrains every day. The project currently exists in a Repo in Databricks. The data scientist has been tasked with improving the feature engineering of the pipeline's preprocessing stage. The data scientist wants to make necessary updates to the code that can be easily adopted into the project without changing what is being run each day.
Which approach should the data scientist take to complete this task?
- A. They can create a new Git repository, import it into Databricks, and copy and paste the existing code from the original repository before making changes.
- B. They can clone the notebooks in the repository into a new Databricks Repo and make the necessary changes.
- C. They can create a new branch in Databricks, commit their changes, and push those changes to the Git provider.
- D. They can clone the notebooks in the repository into a Databricks Workspace folder and make the necessary changes.
正解：C
解説：
The best approach for the data scientist to take in this scenario is to create a new branch in Databricks, commit their changes, and push those changes to the Git provider. This approach allows the data scientist to make updates and improvements to the feature engineering part of the preprocessing pipeline without affecting the main codebase that runs daily. By creating a new branch, they can work on their changes in isolation. Once the changes are ready and tested, they can be merged back into the main branch through a pull request, ensuring a smooth integration process and allowing for code review and collaboration with other team members.
Reference:
Databricks documentation on Git integration: Databricks Repos
質問 # 27
A data scientist has been given an incomplete notebook from the data engineering team. The notebook uses a Spark DataFrame spark_df on which the data scientist needs to perform further feature engineering. Unfortunately, the data scientist has not yet learned the PySpark DataFrame API.
Which of the following blocks of code can the data scientist run to be able to use the pandas API on Spark?
- A. spark_df.to_pandas()
- B. import pyspark.pandas as ps
df = ps.to_pandas(spark_df) - C. import pandas as pd
df = pd.DataFrame(spark_df) - D. import pyspark.pandas as ps
df = ps.DataFrame(spark_df)
正解：D
解説：
To use the pandas API on Spark, the data scientist can run the following code block:
import pyspark.pandas as ps df = ps.DataFrame(spark_df)
This code imports the pandas API on Spark and converts the Spark DataFrame spark_df into a pandas-on-Spark DataFrame, allowing the data scientist to use familiar pandas functions for further feature engineering.
Reference:
Databricks documentation on pandas API on Spark: pandas API on Spark
質問 # 28
A machine learning engineer is converting a decision tree from sklearn to Spark ML. They notice that they are receiving different results despite all of their data and manually specified hyperparameter values being identical.
Which of the following describes a reason that the single-node sklearn decision tree and the Spark ML decision tree can differ?
- A. Spark ML decision trees test binned features values as representative split candidates
- B. Spark ML decision trees test a random sample of feature variables in the splitting algorithm
- C. Spark ML decision trees test more split candidates in the splitting algorithm
- D. Spark ML decision trees automatically prune overfit trees
- E. Spark ML decision trees test every feature variable in the splitting algorithm
正解：A
解説：
One reason that results can differ between sklearn and Spark ML decision trees, despite identical data and hyperparameters, is that Spark ML decision trees test binned feature values as representative split candidates. Spark ML uses a method called "quantile binning" to reduce the number of potential split points by grouping continuous features into bins. This binning process can lead to different splits compared to sklearn, which tests all possible split points directly. This difference in the splitting algorithm can cause variations in the resulting trees.
Reference:
Spark MLlib Documentation (Decision Trees and Quantile Binning).
質問 # 29
An organization is developing a feature repository and is electing to one-hot encode all categorical feature variables. A data scientist suggests that the categorical feature variables should not be one-hot encoded within the feature repository.
Which of the following explanations justifies this suggestion?
- A. One-hot encoding is computationally intensive and should only be performed on small samples of training sets for individual machine learning problems.
- B. One-hot encoding is not a common strategy for representing categorical feature variables numerically.
- C. One-hot encoding is dependent on the target variable's values which differ for each application.
- D. One-hot encoding is not supported by most machine learning libraries.
- E. One-hot encoding is a potentially problematic categorical variable strategy for some machine learning algorithms.
正解：E
解説：
One-hot encoding transforms categorical variables into a format that can be provided to machine learning algorithms to better predict the output. However, when done prematurely or universally within a feature repository, it can be problematic:
Dimensionality Increase: One-hot encoding significantly increases the feature space, especially with high cardinality features, which can lead to high memory consumption and slower computation.
Model Specificity: Some models handle categorical variables natively (like decision trees and boosting algorithms), and premature one-hot encoding can lead to inefficiency and loss of information (e.g., ordinal relationships).
Sparse Matrix Issue: It often results in a sparse matrix where most values are zero, which can be inefficient in both storage and computation for some algorithms.
Generalization vs. Specificity: Encoding should ideally be tailored to specific models and use cases rather than applied generally in a feature repository.
Reference
"Feature Engineering and Selection: A Practical Approach for Predictive Models" by Max Kuhn and Kjell Johnson (CRC Press, 2019).
質問 # 30
What is the name of the method that transforms categorical features into a series of binary indicator feature variables?
- A. Target encoding
- B. String indexing
- C. One-hot encoding
- D. Leave-one-out encoding
- E. Categorical
正解：C
解説：
The method that transforms categorical features into a series of binary indicator variables is known as one-hot encoding. This technique converts each categorical value into a new binary column, which is essential for models that require numerical input. One-hot encoding is widely used because it helps to handle categorical data without introducing a false ordinal relationship among categories.
Reference:
Feature Engineering Techniques (One-Hot Encoding).
質問 # 31
A data scientist uses 3-fold cross-validation when optimizing model hyperparameters for a regression problem. The following root-mean-squared-error values are calculated on each of the validation folds:
* 10.0
* 12.0
* 17.0
Which of the following values represents the overall cross-validation root-mean-squared error?
- A. 13.0
- B. 17.0
- C. 12.0
- D. 39.0
- E. 10.0
正解：A
解説：
To calculate the overall cross-validation root-mean-squared error (RMSE), you average the RMSE values obtained from each validation fold. Given the RMSE values of 10.0, 12.0, and 17.0 for the three folds, the overall cross-validation RMSE is calculated as the average of these three values:
Overall CV RMSE=10.0+12.0+17.03=39.03=13.0Overall CV RMSE=310.0+12.0+17.0=339.0=13.0 Thus, the correct answer is 13.0, which accurately represents the average RMSE across all folds.
Reference:
Cross-validation in Regression (Understanding Cross-Validation Metrics).
質問 # 32
A data scientist is using the following code block to tune hyperparameters for a machine learning model:
Which change can they make the above code block to improve the likelihood of a more accurate model?
- A. Increase num_evals to 100
- B. Change sparkTrials() to Trials()
- C. Change fmin() to fmax()
- D. Change tpe.suggest to random.suggest
正解：A
解説：
To improve the likelihood of a more accurate model, the data scientist can increase num_evals to 100. Increasing the number of evaluations allows the hyperparameter tuning process to explore a larger search space and evaluate more combinations of hyperparameters, which increases the chance of finding a more optimal set of hyperparameters for the model.
Reference:
Databricks documentation on hyperparameter tuning: Hyperparameter Tuning
質問 # 33
Which of the following tools can be used to distribute large-scale feature engineering without the use of a UDF or pandas Function API for machine learning pipelines?
- A. Keras
- B. PyTorch
- C. Scikit-learn
- D. Spark ML
正解：D
解説：
Spark MLlib is a machine learning library within Apache Spark that provides scalable and distributed machine learning algorithms. It is designed to work with Spark DataFrames and leverages Spark's distributed computing capabilities to perform large-scale feature engineering and model training without the need for user-defined functions (UDFs) or the pandas Function API. Spark MLlib provides built-in transformations and algorithms that can be applied directly to large datasets.
Reference:
Databricks documentation on Spark MLlib: Spark MLlib
質問 # 34
A machine learning engineer wants to parallelize the training of group-specific models using the Pandas Function API. They have developed the train_model function, and they want to apply it to each group of DataFrame df.
They have written the following incomplete code block:
Which of the following pieces of code can be used to fill in the above blank to complete the task?
- A. predict
- B. groupedApplyIn
- C. applyInPandas
- D. train_model
- E. mapInPandas
正解：E
解説：
The function mapInPandas in the PySpark DataFrame API allows for applying a function to each partition of the DataFrame. When working with grouped data, groupby followed by applyInPandas is the correct approach to apply a function to each group as a separate Pandas DataFrame. However, if the function should apply across each partition of the grouped data rather than on each individual group, mapInPandas would be utilized. Since the code snippet indicates the use of groupby, the intent seems to be to apply train_model on each group specifically, which aligns with applyInPandas. Thus, applyInPandas is a better fit to ensure that each group generated by groupby is processed through the train_model function, preserving the partitioning and grouping integrity.
Reference
PySpark Documentation on applying functions to grouped data: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.GroupedData.applyInPandas.html
質問 # 35
A data scientist uses 3-fold cross-validation and the following hyperparameter grid when optimizing model hyperparameters via grid search for a classification problem:
* Hyperparameter 1: [2, 5, 10]
* Hyperparameter 2: [50, 100]
Which of the following represents the number of machine learning models that can be trained in parallel during this process?
- A. 0
- B. 1
- C. 2
- D. 3
正解：A
解説：
To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes:
Hyperparameter 1: [2, 5, 10] (3 values)
Hyperparameter 2: [50, 100] (2 values)
The total number of combinations is the product of the number of values for each hyperparameter: 3 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=63 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=6 With 3-fold cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will be: 6 (combinations)×3 (folds)=186 (combinations)×3 (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation. Therefore, 6 models can be trained in parallel.
Reference:
Databricks documentation on hyperparameter tuning: Hyperparameter Tuning
質問 # 36
A data scientist is performing hyperparameter tuning using an iterative optimization algorithm. Each evaluation of unique hyperparameter values is being trained on a single compute node. They are performing eight total evaluations across eight total compute nodes. While the accuracy of the model does vary over the eight evaluations, they notice there is no trend of improvement in the accuracy. The data scientist believes this is due to the parallelization of the tuning process.
Which change could the data scientist make to improve their model accuracy over the course of their tuning process?
- A. Change the number of compute nodes to be half or less than half of the number of evaluations.
- B. Change the number of compute nodes and the number of evaluations to be much larger but equal.
- C. Change the number of compute nodes to be double or more than double the number of evaluations.
- D. Change the iterative optimization algorithm used to facilitate the tuning process.
正解：D
解説：
The lack of improvement in model accuracy across evaluations suggests that the optimization algorithm might not be effectively exploring the hyperparameter space. Iterative optimization algorithms like Tree-structured Parzen Estimators (TPE) or Bayesian Optimization can adapt based on previous evaluations, guiding the search towards more promising regions of the hyperparameter space.
Changing the optimization algorithm can lead to better utilization of the information gathered during each evaluation, potentially improving the overall accuracy.
Reference:
Hyperparameter Optimization with Hyperopt
質問 # 37
A data scientist is developing a machine learning pipeline using AutoML on Databricks Machine Learning.
Which of the following steps will the data scientist need to perform outside of their AutoML experiment?
- A. Model deployment
- B. Model evaluation
- C. Model tuning
- D. Exploratory data analysis
正解：D
解説：
AutoML platforms, such as the one available in Databricks Machine Learning, streamline various stages of the machine learning pipeline including feature engineering, model selection, hyperparameter tuning, and model evaluation. However, exploratory data analysis (EDA) is typically performed outside the AutoML process. EDA involves understanding the dataset, visualizing distributions, identifying anomalies, and gaining insights into data before feeding it into a machine learning pipeline. This step is crucial for ensuring that the data is clean and suitable for model training but is generally done manually by the data scientist.
Reference
Databricks documentation on AutoML: https://docs.databricks.com/applications/machine-learning/automl.html
質問 # 38
Which of the following approaches can be used to view the notebook that was run to create an MLflow run?
- A. Click the "Source" link in the row corresponding to the run in the MLflow experiment page
- B. Open the MLmodel artifact in the MLflow run paqe
- C. Click the "Models" link in the row corresponding to the run in the MLflow experiment paqe
- D. Click the "Start Time" link in the row corresponding to the run in the MLflow experiment page
正解：A
解説：
To view the notebook that was run to create an MLflow run, you can click the "Source" link in the row corresponding to the run in the MLflow experiment page. The "Source" link provides a direct reference to the source notebook or script that initiated the run, allowing you to review the code and methodology used in the experiment. This feature is particularly useful for reproducibility and for understanding the context of the experiment.
Reference:
MLflow Documentation (Viewing Run Sources and Notebooks).
質問 # 39
Which of the following describes the relationship between native Spark DataFrames and pandas API on Spark DataFrames?
- A. pandas API on Spark DataFrames are less mutable versions of Spark DataFrames
- B. pandas API on Spark DataFrames are made up of Spark DataFrames and additional metadata
- C. pandas API on Spark DataFrames are unrelated to Spark DataFrames
- D. pandas API on Spark DataFrames are more performant than Spark DataFrames
- E. pandas API on Spark DataFrames are single-node versions of Spark DataFrames with additional metadata
正解：B
解説：
Pandas API on Spark (previously known as Koalas) provides a pandas-like API on top of Apache Spark. It allows users to perform pandas operations on large datasets using Spark's distributed compute capabilities. Internally, it uses Spark DataFrames and adds metadata that facilitates handling operations in a pandas-like manner, ensuring compatibility and leveraging Spark's performance and scalability.
Reference
pandas API on Spark documentation: https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html
質問 # 40
A machine learning engineer wants to parallelize the inference of group-specific models using the Pandas Function API. They have developed the apply_model function that will look up and load the correct model for each group, and they want to apply it to each group of DataFrame df.
They have written the following incomplete code block:
Which piece of code can be used to fill in the above blank to complete the task?
- A. mapInPandas
- B. predict
- C. groupedApplyInPandas
- D. applyInPandas
正解：D
解説：
To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use the applyInPandas function. This function allows you to apply a Python function on each group of a DataFrame and return a DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance.
prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) ) In this code:
groupby("device_id"): Groups the DataFrame by the "device_id" column.
applyInPandas(apply_model, schema=apply_return_schema): Applies the apply_model function to each group and specifies the schema of the return DataFrame.
Reference:
PySpark Pandas UDFs Documentation
質問 # 41
In which of the following situations is it preferable to impute missing feature values with their median value over the mean value?
- A. When the features contain a lot of extreme outliers
- B. When the features contain no outliers
- C. When the features are of the boolean type
- D. When the features contain no missing no values
- E. When the features are of the categorical type
正解：A
解説：
Imputing missing values with the median is often preferred over the mean in scenarios where the data contains a lot of extreme outliers. The median is a more robust measure of central tendency in such cases, as it is not as heavily influenced by outliers as the mean. Using the median ensures that the imputed values are more representative of the typical data point, thus preserving the integrity of the dataset's distribution. The other options are not specifically relevant to the question of handling outliers in numerical data.
Reference:
Data Imputation Techniques (Dealing with Outliers).
質問 # 42
The implementation of linear regression in Spark ML first attempts to solve the linear regression problem using matrix decomposition, but this method does not scale well to large datasets with a large number of variables.
Which of the following approaches does Spark ML use to distribute the training of a linear regression model for large data?
- A. Spark ML cannot distribute linear regression training
- B. Logistic regression
- C. Singular value decomposition
- D. Iterative optimization
- E. Least-squares method
正解：D
解説：
For large datasets with many variables, Spark ML distributes the training of a linear regression model using iterative optimization methods. Specifically, Spark ML employs algorithms such as Gradient Descent or L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) to iteratively minimize the loss function. These iterative methods are suitable for distributed computing environments and can handle large-scale data efficiently by partitioning the data across nodes in a cluster and performing parallel updates.
Reference:
Spark MLlib Documentation (Linear Regression with Iterative Optimization).
質問 # 43
Which of the following evaluation metrics is not suitable to evaluate runs in AutoML experiments for regression problems?
- A. MSE
- B. F1
- C. R-squared
- D. MAE
正解：B
解説：
The code block provided by the machine learning engineer will perform the desired inference when the Feature Store feature set was logged with the model at model_uri. This ensures that all necessary feature transformations and metadata are available for the model to make predictions. The Feature Store in Databricks allows for seamless integration of features and models, ensuring that the required features are correctly used during inference.
Reference:
Databricks documentation on Feature Store: Feature Store in Databricks
質問 # 44
Which of the following machine learning algorithms typically uses bagging?
- A. IGradient boosted trees
- B. Random forest
- C. Decision tree
- D. K-means
正解：B
解説：
Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging is a technique that involves training multiple base models (such as decision trees) on different subsets of the data and then combining their predictions to improve overall model performance. Each subset is created by randomly sampling with replacement from the original dataset. The Random Forest algorithm builds multiple decision trees and merges them to get a more accurate and stable prediction.
Reference:
Databricks documentation on Random Forest: Random Forest in Spark ML
質問 # 45
A data scientist has written a data cleaning notebook that utilizes the pandas library, but their colleague has suggested that they refactor their notebook to scale with big data.
Which of the following approaches can the data scientist take to spend the least amount of time refactoring their notebook to scale with big data?
- A. They can refactor their notebook to utilize the pandas API on Spark.
- B. They can refactor their notebook to process the data in parallel.
- C. They can refactor their notebook to use the PySpark DataFrame API.
- D. They can refactor their notebook to use the Scala Dataset API.
- E. They can refactor their notebook to use Spark SQL.
正解：A
解説：
The data scientist can refactor their notebook to utilize the pandas API on Spark (now known as pandas on Spark, formerly Koalas). This allows for the least amount of changes to the existing pandas-based code while scaling to handle big data using Spark's distributed computing capabilities. pandas on Spark provides a similar API to pandas, making the transition smoother and faster compared to completely rewriting the code to use PySpark DataFrame API, Scala Dataset API, or Spark SQL.
Reference:
Databricks documentation on pandas API on Spark (formerly Koalas).
質問 # 46
A machine learning engineer is using the following code block to scale the inference of a single-node model on a Spark DataFrame with one million records:
Assuming the default Spark configuration is in place, which of the following is a benefit of using an Iterator?
- A. The model only needs to be loaded once per executor rather than once per batch during the inference process
- B. The model will be limited to a single executor preventing the data from being distributed
- C. The data will be limited to a single executor preventing the model from being loaded multiple times
- D. The data will be distributed across multiple executors during the inference process
正解：A
解説：
Using an iterator in the pandas_udf ensures that the model only needs to be loaded once per executor rather than once per batch. This approach reduces the overhead associated with repeatedly loading the model during the inference process, leading to more efficient and faster predictions. The data will be distributed across multiple executors, but each executor will load the model only once, optimizing the inference process.
Reference:
Databricks documentation on pandas UDFs: Pandas UDFs
質問 # 47
......
無料Databricks-Machine-Learning-Associate試験問題集試験点数を伸ばそう：https://jp.fast2test.com/Databricks-Machine-Learning-Associate-premium-file.html