Uma lista de funções de custo usadas em redes neurais, juntamente com aplicativos

133

Quais são as funções de custo comuns usadas na avaliação do desempenho de redes neurais?

Detalhes

(fique à vontade para pular o restante desta pergunta, minha intenção aqui é simplesmente fornecer esclarecimentos sobre a notação que as respostas podem usar para ajudá-las a serem mais compreensíveis para o leitor em geral)

Eu acho que seria útil ter uma lista de funções de custo comuns, juntamente com algumas maneiras pelas quais elas foram usadas na prática. Portanto, se outras pessoas estão interessadas nisso, acho que um wiki da comunidade é provavelmente a melhor abordagem, ou podemos removê-lo se não estiver relacionado ao tópico.

Notação

Então, para começar, gostaria de definir uma notação que todos usamos ao descrevê-las, para que as respostas se encaixem bem.

Esta notação é do livro de Neilsen .

Uma Rede Neural Feedforward é formada por várias camadas de neurônios conectados. Em seguida, recebe uma entrada, que "escorre" pela rede e, em seguida, a rede neural retorna um vetor de saída.

Mais formalmente, chamar a activação (aka saída) do neurónio no camada, onde é a elemento no vector de entrada.ajijthithaj1jth

Em seguida, podemos relacionar a entrada da próxima camada com a anterior por meio da seguinte relação:

aji=σ(k(wjkiaki1)+bji)

Onde

σ é a função de ativação,

wjki é o peso do neurônio na camada para o neurônio na camada ,kth(i1)thjthith

bji is the bias of the jth neuron in the ith layer, and

aji represents the activation value of the jth neuron in the ith layer.

Sometimes we write zji to represent k(wjkiaki1)+bji, in other words, the activation value of a neuron before applying the activation function.

enter image description here

For more concise notation we can write

ai=σ(wi×ai1+bi)

To use this formula to compute the output of a feedforward network for some input IRn, set a1=I, then compute a2, a3, ...,am, where m is the number of layers.

Introduction

A cost function is a measure of "how good" a neural network did with respect to it's given training sample and the expected output. It also may depend on variables such as weights and biases.

A cost function is a single value, not a vector, because it rates how good the neural network did as a whole.

Specifically, a cost function is of the form

C(W,B,Sr,Er)

where W is our neural network's weights, B is our neural network's biases, Sr is the input of a single training sample, and Er is the desired output of that training sample. Note this function can also potentially be dependent on yji and zji for any neuron j in layer i, because those values are dependent on W, B, and Sr.

In backpropagation, the cost function is used to compute the error of our output layer, δL, via

δjL=CajLσ(zji)
.

Which can also be written as a vector via

δL=aCσ(zi)
.

We will provide the gradient of the cost functions in terms of the second equation, but if one wants to prove these results themselves, using the first equation is recommended because it's easier to work with.

Cost function requirements

To be used in backpropagation, a cost function must satisfy two properties:

1: The cost function C must be able to be written as an average

C=1nxCx

over cost functions Cx for individual training examples, x.

This is so it allows us to compute the gradient (with respect to weights and biases) for a single training example, and run Gradient Descent.

2: The cost function C must not be dependent on any activation values of a neural network besides the output values aL.

Technically a cost function can be dependent on any aji or zji. We just make this restriction so we can backpropagte, because the equation for finding the gradient of the last layer is the only one that is dependent on the cost function (the rest are dependent on the next layer). If the cost function is dependent on other activation layers besides the output one, backpropagation will be invalid because the idea of "trickling backwards" no longer works.

Also, activation functions are required to have an output 0ajL1 for all j. Thus these cost functions need to only be defined within that range (for example, ajL is valid since we are guaranteed ajL0).

Phylliida
fonte
3
This is a Q&A site, and the format of this post doesn't really fit that. You should probably put the majority of the content in an answer, and leave just the question (e.g. What is a list of cost functions used in NNs?).
Roger Fan
Okay, is that better? I think the definitions are important otherwise the answers become vague for those that aren't familiar with the terminology the writer uses.
Phylliida
But what if a different answer uses different notation or terminology?
Roger Fan
3
The idea is that everyone uses the same terminology here, and that if it's different we convert it to this, so the answers "fit" with each other. But I suppose I could remove that piece if you don't think it's helpful.
Phylliida
1
I just think the detail the question goes into isn't really necessary or relevant. It seems a bit excessive and limiting, but that's just me.
Roger Fan

Respostas:

85

Here are those I understand so far. Most of these work best when given values between 0 and 1.

Quadratic cost

Also known as mean squared error, maximum likelihood, and sum squared error, this is defined as:

CMST(W,B,Sr,Er)=0.5j(ajLEjr)2

The gradient of this cost function with respect to the output of a neural network and some sample r is:

aCMST=(aLEr)

Cross-entropy cost

Also known as Bernoulli negative log-likelihood and Binary Cross-Entropy

CCE(W,B,Sr,Er)=j[Ejr ln ajL+(1Ejr) ln (1ajL)]

The gradient of this cost function with respect to the output of a neural network and some sample r is:

aCCE=(aLEr)(1aL)(aL)

Exponentional cost

This requires choosing some parameter τ that you think will give you the behavior you want. Typically you'll just need to play with this until things work good.

CEXP(W,B,Sr,Er)=τ exp(1τj(ajLEjr)2)

where exp(x) is simply shorthand for ex.

The gradient of this cost function with respect to the output of a neural network and some sample r is:

aC=2τ(aLEr)CEXP(W,B,Sr,Er)

I could rewrite out CEXP, but that seems redundant. Point is the gradient computes a vector and then multiplies it by CEXP.

Hellinger distance

CHD(W,B,Sr,Er)=12j(ajLEjr)2

You can find more about this here. This needs to have positive values, and ideally values between 0 and 1. The same is true for the following divergences.

The gradient of this cost function with respect to the output of a neural network and some sample r is:

aC=aLEr2aL

Kullback–Leibler divergence

Also known as Information Divergence, Information Gain, Relative entropy, KLIC, or KL Divergence (See here).

Kullback–Leibler divergence is typically denoted

DKL(PQ)=iP(i)lnP(i)Q(i)
,

where DKL(PQ) is a measure of the information lost when Q is used to approximate P. Thus we want to set P=Ei and Q=aL, because we want to measure how much information is lost when we use aji to approximate Eji. This gives us

CKL(W,B,Sr,Er)=jEjrlogEjrajL

The other divergences here use this same idea of setting P=Ei and Q=aL.

The gradient of this cost function with respect to the output of a neural network and some sample r is:

aC=EraL

Generalized Kullback–Leibler divergence

From here.

CGKL(W,B,Sr,Er)=jEjrlogEjrajLj(Ejr)+j(ajL)

The gradient of this cost function with respect to the output of a neural network and some sample r is:

aC=aLEraL

Itakura–Saito distance

Also from here.

CGKL(W,B,Sr,Er)=j(EjrajLlogEjrajL1)

The gradient of this cost function with respect to the output of a neural network and some sample r is:

aC=aLEr(aL)2

Where ((aL)2)j=ajLajL. In other words, (aL)2 is simply equal to squaring each element of aL.

Phylliida
fonte
Thanks for sharing, you can also consider these: github.com/torch/nn/blob/master/doc/criterion.md
Yannis Assael
2
you have a small mistake in the denominator of the cross-entropy derivative, it should be a*(1-a) not a*(1+a)
Amro
1
It would also be cool to show the pinball loss function to minimize error quantiles rather than average error. Very used in decision support systems.
Ricardo Cruz
where can I see graphs for these?
coiso
1
With regards to the Quadratic Cost Function, you should note that "mean squared error" "maximum likelihood" "sum squared error". Authors may use the name (incorrectly) interchangeably, but they are not the same thing.
Jon
20

Don't have the reputation to comment, but there are sign errors in those last 3 gradients.

In the KL divergence,

C=jEjlog(Ej/aj)=jEjlog(Ej)Ejlog(aj)dC=jEjdlog(aj)=j(Ej/aj)dajaC=Ea
This same sign error appears in the Generalized KL divergence.

In the Itakura-Saito distance ,

C=j(Ej/aj)log(Ej/aj)1=j(Ej/aj)log(Ej)+log(aj)1dC=j(Ej/aj2)daj+dlog(aj)=j(1/aj)daj(Ej/aj2)daj=j(ajEj)/aj2dajaC=aE(a)2
frank
fonte