如何在Keras或Theano中实现具有指数衰减学习率的卷积神经网络

2019年7月28日 451次阅读

我想在Keras或Theano中实现具有指数衰减学习率的卷积神经网络(CNN).学习率根据以下更新法动态更改：

eta = et0*exp(LossFunction)
where et0 is the initial learning rate and LossFunction is a cost function

我知道Keras允许设置SGD优化器：

SGD(lr, momentum0, decay, nesterov)

衰减项仅允许在每个时期内固定的衰减学习率衰减.

如何使用在成本函数方面呈指数衰减的学习率来设置或编码SGD？为了您的信息,我在Keras发布了SGD的源代码：

class SGD(Optimizer):

'''Stochastic gradient descent, with support for momentum,
learning rate decay, and Nesterov momentum.


# Arguments
    lr: float >= 0. Learning rate.
    momentum: float >= 0. Parameter updates momentum.
    decay: float >= 0. Learning rate decay over each update.
    nesterov: boolean. Whether to apply Nesterov momentum.
'''

def __init__(self, lr=0.01, momentum=0., decay=0.,

             nesterov=False, **kwargs):

    super(SGD, self).__init__(**kwargs)
    self.__dict__.update(locals())
    self.iterations = K.variable(0.)
    self.lr = K.variable(lr)
    self.momentum = K.variable(momentum)
    self.decay = K.variable(decay)
    self.inital_decay = decay

def get_updates(self, params, constraints, loss):
    grads = self.get_gradients(loss, params)
    self.updates = []

    lr = self.lr
    if self.inital_decay > 0:
        lr *= (1. / (1. + self.decay * self.iterations))
        self.updates .append(K.update_add(self.iterations, 1))

    # momentum
    shapes = [K.get_variable_shape(p) for p in params]
    moments = [K.zeros(shape) for shape in shapes]
    self.weights = [self.iterations] + moments

    for p, g, m in zip(params, grads, moments):
        v = self.momentum * m - lr * g  # velocity
        self.updates.append(K.update(m, v))

        if self.nesterov:
            new_p = p + self.momentum * v - lr * g
        else:
            new_p = p + v

        # apply constraints
        if p in constraints:
            c = constraints[p]
            new_p = c(new_p)

        self.updates.append(K.update(p, new_p))
    return self.updates

def get_config(self):
    config = {'lr': float(K.get_value(self.lr)),
              'momentum': float(K.get_value(self.momentum)),
              'decay': float(K.get_value(self.decay)),
              'nesterov': self.nesterov}

    base_config = super(SGD, self).get_config()
    return dict(list(base_config.items()) + list(config.items()))

最佳答案 我认为你可以使用以下架构获得行为：

>使用this创建一个新的学习速率控制器类.
>使其成为接受训练集并在提供适合方法时开始学习率的构造函数.
>让它计算每个纪元后的损失并更新学习率.

点赞