Wednesday, 3 April 2013

probability - Expected value of applying the sigmoid function to a normal distribution



Short version:




I would like to calculate the expected value if you apply the sigmoid function 11+ex to a normal distribution with expected value μ and standard deviation σ.



If I'm correct this corresponds to the following integral:



11+ex1σ2π e(xμ)22σ2dx



However, I can't solve this integral. I've tried manually, with Maple and with Wolfram|Alpha, but didn't get anywhere.



Some background info (why I want to do this):




Sigmoid functions are used in artificial neural networks as an activation function, mapping a value of (,) to (0,1). Often this value is used directly in further calculations but sometimes (e.g. in RBM's) it's first stochastically rounded to a 0 or a 1, with the probabililty of a 1 being that value. The stochasticity helps the learning, but is sometimes not desired when you finally use the network. Just using the normal non-stochastic methods on a network that you trained stochastically doesn't work though. It changes the expected result, because (in short):



E[S(X)]S(E[X])



for most X. However, if you approximate X as a normal distribution and could somehow calculate this expected value, you could eliminate most of the bias. That's what I'm trying to do.


Answer



I doubt that there's a closed-form solution. However, here's a series in powers of σ:



(eμ+1)1+(eμ1)eμ2(eμ+1)3σ2+(e3μ11e2μ+11eμ1)eμ8(eμ+1)5σ4+eμ(e5μ57e4μ+302e3μ302e2μ+57eμ1)48(eμ+1)7σ6+eμ(e7μ247e6μ+4293e5μ15619e4μ+15619e3μ4293e2μ+247eμ1)384(eμ+1)9σ8+O(σ10)



EDIT: To obtain this, first do the change of variables x=μ+σt. The
integral becomes
12πet2/21+eμσt dt

Now take the Maclaurin series 11+eμσt=11+eμ+eμσt(1+eμ)2+eμ(eμ1)σ2t2(1+eμ)3+
and integrate term by term.


No comments:

Post a Comment

real analysis - How to find limhrightarrow0fracsin(ha)h

How to find lim without lhopital rule? I know when I use lhopital I easy get $$ \lim_{h\rightarrow 0}...