My Math Forum  

Go Back   My Math Forum > College Math Forum > Calculus

Calculus Calculus Math Forum

LinkBack Thread Tools Display Modes
March 19th, 2017, 03:12 AM   #1
Joined: Mar 2017
From: uk

Posts: 1
Thanks: 0

Derivative Mind-bending!

Hi all,

I've not touched calculus post my maths degree many years ago.

I need to calculate a gradient, to use in back-propagation in a neural network, but it's making my head hurt!

I have the formula

δ(t)  = tanh[ <w, f(t) > + b + uδ(t−1) ]
where t, t-1 represent time steps

and need to calculate

dδ(t)/dθ = ∂δ(t)/∂θ + ∂δ(t)/∂δ(t−1) * dδ(t-1)/dθ
where θ comprises the parameter set [w, b, u]. b and u are scalars, but w is a vector of multiple elements, e.g. w1, w2 etc..

The derivative of tanh(x) is 1 - tanh^2(x), so I assume the partial derivative,

∂δ(t)/dδ(t−1) = 1 - tanh^2[ <w, f(t) > + b + uδ(t−1) ]
The final element (dδ(t-1)/dθ) comes from recursively calculating the gradient, so I'm happy with that.

I'm less sure about the first element, given the θ decomposition into its constituent elements.

Any ideas?!

Last edited by skipjack; March 19th, 2017 at 04:00 PM.
mctaff is offline  

  My Math Forum > College Math Forum > Calculus

derivative, mindbending

Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
Bending stresses greeny93 Physics 0 December 11th, 2016 09:54 AM
hydro-bending setup sarah208 Trigonometry 3 January 6th, 2015 11:38 PM
another mind twister mathkid Calculus 9 November 27th, 2012 07:58 PM
never mind, sorry ElMarsh Real Analysis 1 October 24th, 2009 06:32 PM
never mind - i got it tomotomo Algebra 1 July 30th, 2008 05:37 AM

Copyright © 2018 My Math Forum. All rights reserved.