By Wei-Chung Allen Lee, Hayden Huang, Guoping Feng, Joshua R. Sanes, Emery N. Brown, Peter T. So, Elly Nedivi - Dynamic Remodeling of Dendritic Arbors in GABAergic Interneurons of Adult Visual Cortex. Lee WCA, Huang H, Feng G, Sanes JR, Brown EN, et al. PLoS Biology Vol. 4, No. 2, e29. doi:10.1371/journal.pbio.0040029, Figure 6f, slightly altered (plus scalebar, minus letter "f".), CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=3482256
Simulating a neuron with a simple "activation function" like TANH or RELU is just plain silly.
Error backpropagation works great but gradient sharing across the network does not scale.
We began with simple neural nets tuned by an evolutionary algorithm. We built the Evolutionary Compute Platform so we could scale at low cost. We're now working on Multi-cellular Reinforcement Learning.
Multi-cellular structure is a prerequisite for harnessing quantum annealing as a replacement for error back-propagation and GPU's.