Tensorflow

There are three ways to create Keras models:

• The Sequential model, which is very straightforward (a simple list of layers), but is limited to single-input, single-output stacks of layers (as the name gives away).

• The Functional API, which is an easy-to-use, fully-featured API that supports arbitrary model architectures. For most people and most use cases, this is what you should be using. This is the Keras "industry strength" model.

• Model Subclassing, where you implement everything from scratch on your own. Use this if you have complex, out-of-the-box research use cases.

Summary

model.summary(line_length=None,
              positions=None,
              print_fn=None)

Prints a string summary of the network.

Arguments

• line_length: Total length of printed lines (e.g. set this to adapt the display to different terminal window sizes).

• positions: Relative or absolute positions of log elements in each line. If not provided, defaults to [.33, .55, .67, 1.].

• print_fn: Print function to use. Defaults to print. It will be called on each line of the summary. You can set it to a custom function in order to capture the string summary.

Raises

• ValueError: if summary() is called before the model is built.

Loss

Binary Crossentropy

Binary Crossentropy / Log Loss measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label. So predicting a probability of .012 when the actual observation label is 1 would be bad and result in a high loss value. A perfect model would have a log loss of 0.

Log Loss

$$BinaryCrossEntropy = -{(y\log(p) + (1 - y)\log(1 - p))}$$

keras.losses.BinaryCrossentropy(from_logits=False,
                                label_smoothing=0,
                                reduction=tf.keras.losses.Reduction.AUTO,
                                name='binary_crossentropy')

from_logits

default: False True: Considers predicted output as tensor of logits False: By default they are assumed to be probablities NOTE: Setting it to True may be more numerically stable

label_smoothing

options: [0,1] 0: no smoothing occurs 1: heavy smoothing def: used to push predicted labels towards 0.5 to avoid overconfidence by restricting one logit value being too bigger than the rest. It's a regularization technique in crossentropy loss family.

reduction

tf.keras.losses.Reduction.SUM tf.keras.losses.Reduction.NONE tf.keras.losses.Reduction.AUTO

Categorical Crossentropy

Use this loss function when there are two or more label classes. We expect labels to be provided in a one_hot representation. If you want to provide labels as integers, please use SparseCategoricalCrossentropy loss. There should be # classes floating point values per feature.

In the snippet below, there is # classes floating pointing values per example. The shape of both y_pred and y_true are [batch_size, num_classes].

tf.keras.losses.CategoricalCrossentropy(from_logits=False,
                                        label_smoothing=0,
                                        reduction=losses_utils.ReductionV2.AUTO,
                                        name='categorical_crossentropy')

$$CategoricalCrossentropy = -\sum_{i=1}^Cy_{o,c}\log(p_{o,c})$$

Sparse Categorical Entropy

Use this loss function when there are two or more label classes. We expect labels to be provided as integers. If you want to provide labels using one-hot representation, please use CategoricalCrossentropy loss. There should be # classes floating point values per feature for y_pred and a single floating point value per feature for y_true.

In the snippet below, there is a single floating point value per example for y_true and # classes floating pointing values per example for y_pred. The shape of y_true is [batch_size] and the shape of y_pred is [batch_size, num_classes].

tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False,
                                              reduction=losses_utils.ReductionV2.AUTO,
                                              name='sparse_categorical_crossentropy')

Categorical Hinge

The hinge loss is used for "maximum-margin" classification, most notably for support vector machines (SVMs).

$$HingeLoss = max(0, 1 - y.y')$$

tf.keras.losses.CategoricalHinge(reduction=losses_utils.ReductionV2.AUTO,
                                 name='categorical_hinge')

Cosine Similarity

$$Cosine Similarity= - sum(y * y')$$

tf.keras.losses.CosineSimilarity(axis=-1,
                                 reduction=losses_utils.ReductionV2.AUTO,
                                 name='cosine_similarity')

Mean Absolute Error / L1

$$Loss = |y - y'|$$

tf.keras.losses.MeanAbsoluteError(reduction=losses_utils.ReductionV2.AUTO,
                                  name='mean_absolute_error')

Mean Absolute Percentage Error

$$Loss = 100 * (|y - y'| / y)$$

tf.keras.losses.MeanAbsolutePercentageError(reduction=losses_utils.ReductionV2.AUTO,
                                            name='mean_absolute_percentage_error')

Mean Squared Error / L2

tf.keras.losses.MeanSquaredError(reduction=losses_utils.ReductionV2.AUTO,
                                 name='mean_squared_error')

$$Loss = |y - y'|^2$$

Optimizers

Adam

Adadelta