Activation Function – A Cubic Hermite Spline ? This is no Sigmoid!

In the midst of writing my semi-neural net compressor I have introduced a scaling function based off the premise of an Activation function for a neural net – stretch and squash. Because I always work within a normalized domain of 0.0 to 1.0 I was looking for a logarithmic type function that would work similar to a sigmoid to bring my probabilities closer to a best guess or at least a better prediction.

Below is a graph showing a hermite basis function of a Cubic Hermit Spline , which acts as a sigmoid in the domain of 0.0 to 1.0. The function is very simple and results are phenomenal. In comparison I show a linear curve, of course with no scaling and a hyperbolic tangent that obviously is skewed to fit within the domain 0.0 to 1.0. I have also tested other functions such as log sigmoid and double exponential sigmoid with decent results. This simple function results in a logarithmic curve that seems to produce amazing results. Of course outside the domain I use, the function is definitely a fake sigmoid.

The formula is very basic: p(x) = -2x^3+3x^2

* ^ = exponent

Activation Functions


This entry was posted in Articles. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *