tf.keras does support Metric classes which can evaludate metrics at each batch. However, it does have a limitation that it can only calculat on training data and it can only output only one value.

This becomes a problem especially in cases such as when the user does validation evaluation and needs to record more than one metrics at once. A callback is more suitable for periodic validation set evaluation since it would be slow and computationally expensive to execute this at every training step. However, a native callback does not solve the problem of saving any calculated values into tfsummary so that the user can track down these values from tensorboard. However this part can be hacked.

First, a vanilla TensorBoard callback needs to be instantiated and passed on to our custom callback.

import tensorflow as tf
from .test_callback import TestCallback


model = tf.keras.Model()

tb_callback = tf.keras.callbacks.TensorBoard("somedir")
test_callback = TestCallback(tb_callback)

callbacks = [
    tb_callback,
    test_callback
]

model.fit(x_data, y_data, batch_size=8, epochs=10, callbacks=callbacks)

The TestCallback looks like this:

import tensorflow as tf

class TestCallback(tf.keras.callbacks.Callback):


    def __init__(self, tb_callback):

        self.tb_callback = tb_callback
        self.step_number =0


    def on_epoch_end(self, epoch, logs=None):

        test_input = "something"
        test_gt = "some ground truth data"

        test_output = self.model.predict(test_input)

        metric1, metric2 = get_metrics(test_gt, test_output)

        items_to_write={
            "metric1_name": metric1,
            "metric2_name": metric2
        }

        writer = self.tb_callback.writer

        for name, value in items_to_write.items():

            summary = tf.summary.Summary()
            summary_value = summary.value.add()
            summary_value.simple_value = value
            summary_value.tag = name
            writer.add_summary(summary, self.step_number)
            writer.flush()

        self.step_number += 1

2 Comments

Federico · June 27, 2020 at 12:09 am

Well, this is the result on my notebook 😛
AttributeError: ‘TensorBoard’ object has no attribute ‘writer’

Cathy · April 4, 2021 at 12:31 am

This is exactly what i’m looking for. Really helps, thanks a lot!

Leave a Reply

Your email address will not be published.