My Math Forum  

Go Back   My Math Forum > College Math Forum > Calculus

Calculus Calculus Math Forum

LinkBack Thread Tools Display Modes
March 19th, 2017, 04:12 AM   #1
Joined: Mar 2017
From: uk

Posts: 1
Thanks: 0

Derivative Mind-bending!

Hi all,

I've not touched calculus post my maths degree many years ago.

I need to calculate a gradient, to use in back-propagation in a neural network, but it's making my head hurt!

I have the formula

δ(t)  = tanh[ <w, f(t) > + b + uδ(t−1) ]
where t, t-1 represent time steps

and need to calculate

dδ(t)/dθ = ∂δ(t)/∂θ + ∂δ(t)/∂δ(t−1) * dδ(t-1)/dθ
where θ comprises the parameter set [w, b, u]. b and u are scalars, but w is a vector of multiple elements, e.g. w1, w2 etc..

The derivative of tanh(x) is 1 - tanh^2(x), so I assume the partial derivative,

∂δ(t)/dδ(t−1) = 1 - tanh^2[ <w, f(t) > + b + uδ(t−1) ]
The final element (dδ(t-1)/dθ) comes from recursively calculating the gradient, so I'm happy with that.

I'm less sure about the first element, given the θ decomposition into its constituent elements.

Any ideas?!

Last edited by skipjack; March 19th, 2017 at 05:00 PM.
mctaff is offline  

  My Math Forum > College Math Forum > Calculus

derivative, mindbending

Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
Bending stresses greeny93 Physics 0 December 11th, 2016 10:54 AM
hydro-bending setup sarah208 Trigonometry 3 January 7th, 2015 12:38 AM
another mind twister mathkid Calculus 9 November 27th, 2012 08:58 PM
never mind, sorry ElMarsh Real Analysis 1 October 24th, 2009 07:32 PM
never mind - i got it tomotomo Algebra 1 July 30th, 2008 06:37 AM

Copyright © 2018 My Math Forum. All rights reserved.