gave tfruns package TensorFlow from R provides a set of tools for tracking, visualizing, and managing training runs and experiments. Use the tfruns package to:
-
Track the hyperparameters, metrics, output, and source code of each training session.
-
Compare the hyperparameters and metrics of all runs to find the best performing model.
-
Automatically generate reports to visualize individual training runs or comparisons between runs.
You can install tfruns The following packages from GitHub:
devtools::install_github("rstudio/tfruns")Full documentation for tfruns is available at TensorFlow for R website.
tfruns Intended to be used with Kira and/or tfestimators packages, which both provide high-level interfaces from R to TensorFlow. These packages can be installed with:
# keras
install.packages("keras")
# tfestimators
devtools::install_github("rstudio/tfestimators")training
The following sections will describe our various capabilities. tfruns. Our example training script (mnist_mlp.R) trains a Keras model to recognize MNIST digits.
To train a model with tfrunsjust use training_run() function in place of source() function to execute your R script. For example:
When training is complete, a summary of the run will automatically appear if you are in an interactive R session:
The metrics and output of each run are automatically captured within a. Run the directory which is unique to each run you start. Note that for Keras and TF estimator models this data is captured automatically (no changes to your source code are required).
can call you latest_run() function to view the results of the last run (including the path to the run directory that stores all output from the run):
$ run_dir : chr "runs/2017-10-02T14-23-38Z"
$ eval_loss : num 0.0956
$ eval_acc : num 0.98
$ metric_loss : num 0.0624
$ metric_acc : num 0.984
$ metric_val_loss : num 0.0962
$ metric_val_acc : num 0.98
$ flag_dropout1 : num 0.4
$ flag_dropout2 : num 0.3
$ samples : int 48000
$ validation_samples: int 12000
$ batch_size : int 128
$ epochs : int 20
$ epochs_completed : int 20
$ metrics : chr "(metrics data frame)"
$ model : chr "(model summary)"
$ loss_function : chr "categorical_crossentropy"
$ optimizer : chr "RMSprop"
$ learning_rate : num 0.001
$ script : chr "mnist_mlp.R"
$ start : POSIXct[1:1], format: "2017-10-02 14:23:38"
$ end : POSIXct[1:1], format: "2017-10-02 14:24:24"
$ completed : logi TRUE
$ output : chr "(script ouptut)"
$ source_code : chr "(source archive)"
$ context : chr "local"
$ type : chr "training"The run directory used in the example above is “runs/2017-10-02T14-23-38Z”. Run directories are by default inside the “runs” subdirectory of the current working directory, and use a timestamp as the name of the run directory. You can view the report for any race using view_run() Function:
view_run("runs/2017-10-02T14-23-38Z")Comparing runs
Let's make some changes to our training script to see if we can improve the performance of the model. We will change the number of units in our first dense layer to 128, change learning_rate 0.001 to 0.003 and run 30 instead of 20. epochs. After making these changes to the source code we run the script again using training_run() as before:
training_run("mnist_mlp.R")It will also show us a report summarizing the results of the run, but what we're really interested in is the comparison between this run and the previous one. We can see a comparison through compare_runs() Function:
The comparison report shows the model properties and metrics side-by-side, as well as differences in the source code and output of the training script.
Remember compare_runs() By default will compare the last two runs, however you can pass any two run directories you want to compare.
Use of flags
Tuning a model often requires exploring the effects of changes in many hyperparameters. The best way to approach this is usually not to change the source code of the training script as we did above, but instead by specifying flags for the key parameters you want to vary. In the example script you can see that we have done this for work. dropout Layers:
FLAGS <- flags(
flag_numeric("dropout1", 0.4),
flag_numeric("dropout2", 0.3)
)These flags are then used in our model definition here:
model <- keras_model_sequential()
model %>%
layer_dense(units = 128, activation = 'relu', input_shape = c(784)) %>%
layer_dropout(rate = FLAGS$dropout1) %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dropout(rate = FLAGS$dropout2) %>%
layer_dense(units = 10, activation = 'softmax')Once we have defined the flags, we can pass the alternate flag values. training_run() as follows:
training_run('mnist_mlp.R', flags = c(dropout1 = 0.2, dropout2 = 0.2))You don't need to specify all flags (any flag omitted will just use its default value).
Flags make it very straightforward to systematically explore the effects of changes in hyperparameters on model performance, for example:
Flag values ​​are automatically added to the run data with a “flag_” prefix (eg flag_dropout1, flag_dropout2).
See the article on Training flags For additional documentation on using flags.
Analyzing runs
We have demonstrated visualizing and comparing one or two runs, however as you collect more runs you will typically want to analyze and compare several runs. You can use ls_runs() function to get a dataframe containing summary information on all the runs you've made within a given directory:
# A tibble: 6 x 27
run_dir eval_loss eval_acc metric_loss metric_acc metric_val_loss
1 runs/2017-10-02T14-56-57Z 0.1263 0.9784 0.0773 0.9807 0.1283
2 runs/2017-10-02T14-56-04Z 0.1323 0.9783 0.0545 0.9860 0.1414
3 runs/2017-10-02T14-55-11Z 0.1407 0.9804 0.0348 0.9914 0.1542
4 runs/2017-10-02T14-51-44Z 0.1164 0.9801 0.0448 0.9882 0.1396
5 runs/2017-10-02T14-37-00Z 0.1338 0.9750 0.1097 0.9732 0.1328
6 runs/2017-10-02T14-23-38Z 0.0956 0.9796 0.0624 0.9835 0.0962
# ... with 21 more variables: metric_val_acc , flag_dropout1 ,
# flag_dropout2 , samples , validation_samples , batch_size ,
# epochs , epochs_completed , metrics , model , loss_function ,
# optimizer , learning_rate , script , start , end ,
# completed , output , source_code , context , type Because ls_runs() Returns a data frame that you can use to render a configurable, filterable version of it in RS Studio. View() Function:
gave ls_runs() function also supports subset And order Arguments For example, all of the following runs will achieve eval accuracy better than 0.98:
ls_runs(eval_acc > 0.98, order = eval_acc)Your results may pass ls_runs() Comparing runs (which will always compare the first two runs passed). For example, it will compare the two runs that performed best in terms of diagnostic accuracy:
compare_runs(ls_runs(eval_acc > 0.98, order = eval_acc))
RStudio IDE
If you use with RStudio. tfrunsit is strongly recommended that you update the current one. Preview release As of RStudio v1.1, there are several integration points with the IDE that this new release requires.
add
gave tfruns The package installs an RStudio IDE addin that provides quick access to frequently used functions from the Addins menu.
Note that you can use tools -> Edit keyboard shortcuts. To assign a keyboard shortcut to one or more Addin commands within RStudio.
Background training
RStudio v1.1 includes a terminal pane along with a console pane. Since training runs can be quite long, it is often useful to run them in the background to keep the R console free for other work. You can do this from the terminal like this:
If you are not running within RStudio you can of course use a system terminal window for background training.
Publishing reports
Training run views and comparisons are HTML documents that can be saved and shared with others. When viewing a report in RStudio v1.1 you can save a copy of the report or publish it to RPubs or RStudio Connect:
If you are not running within RStudio you can use save_run_view() And save_run_comparison() Functions for creating standalone HTML versions of run reports.
Management of runs
Various tools are available for managing training run output, including:
-
Exporting run artifacts (eg saved models).
-
Copying and clearing run directories.
-
Using a custom run directory for an experiment or other set of related runs.
gave Management of runs The article provides additional details on how to use these features.