Diary/2019-3-11

R-Wonc'19

というのを聴講．
https://usability-research.r-ccs.riken.jp/r-wonc19/

Giacomo Indiveri, Neural processing and learning electronic circuits for building neuromorphic cognitive agents

• intro
  • driven by AI
  • driven by Big-Data https://www.nature.com/articles/d41586-018-01683-1
• Bee brain specs
  • 1mg, 1mm^3, 960,000 neurons, 10^{-15} J/spike energy/op
• Neuromorphic processor chips
  • spikes in, and spikes out
  • analog subthreshold circuits
  • inhomogeneous, imprecise, and noisy,
  • massively parallel
• DYNAP-SEL-Dynamic Neuromorphic Asynch Processor with Self Learning
• Adaptive CardioRespiratory Pacemaker EU project

Mike Davies, A New Era of Neuromorphic Computing

• cf. https://newsroom.intel.com/editorials/intel-creates-neuromorphic-research-community/
• Background (pict.)
• The Engineering Perspective (pict.)
• LOIHI: https://www.computer.org/csdl/magazine/mi/2018/01/mmi2018010082/13rRUIJcWtw
  • https://www.researchgate.net/publication/322548911_Loihi_A_Neuromorphic_Manycore_Processor_with_On-Chip_Learning
  • 128 neuromorphic cores, 128k neurons, 128M synapses
  • 14nm FinFET
  • cf. http://niceworkshop.org/wp-content/uploads/2018/05/Mike-Davies-NICE-Loihi-Intro-Talk-2018.pdf
• Loihi systems
  • wolf mountain, Nahuku, Kapoho Bay, ...
    • cf. https://converge360.com/Blogs/Future-Tech-Blog/2018/12/NeuroBiological-USB-Intel.aspx
    • cf. https://www.top500.org/news/intel-ramps-up-neuromorphic-computing-effort-with-new-research-partners/
• SNN algorithms discovery and development
• Speech Recognition: Keyword spotting
  • https://arxiv.org/abs/1812.01739
• Spiking LCA dynamics
• Spike-based LSTMs LSNN
  • cf. Long short-term memory and learning-to-learn in networks of spiking neurons https://arxiv.org/abs/1803.09574
  • cf. Biologically inspired alternatives to back-propagation through time for learning in recurrent neural nets https://arxiv.org/abs/1901.09049
  • cf. Adaptive Control of a Robot Arm using Loihi https://royalsocietypublishing.org/doi/full/10.1098/rspb.2016.2134
• Graph search
• Olfacton-Inspired One Shot Learning
  • cf. olfaction inspired machine learning https://arxiv.org/pdf/1802.05405.pdf
• Why Spikes: (pic.)

Cerebellum

• a little brain
  • cf. Cerebellar ataxia - https://www.youtube.com/watch?v=Txlvuu2byUY
• simulation by CPU, GPU, FPGA, PEZY
  • GPU, FPGA, PEZY - realtime simulation
• human-scale cerebellum on K
  • 68 billion neurons on 82,944 nodes
  • MONET (in-house simulator)
  • 600 time slower than realtime (10min. for 1s)
• application
  • arm control
  • reinforcement learning in cerebellum
    • reinforcement learning can go in parallel (massively)
    • - cf. Hybrid Reward Architecture for Reinforcement Learning https://arxiv.org/abs/1706.04208
    • - cf. Hybrid Reward Architecture https://github.com/Maluuba/hra

A Benchmarking and Programming Framework for Spiking Neuromorphic Computing Systems

A Survey of Neuromorphic Computing and Neural Networks in Hardware - https://arxiv.org/abs/1705.06963

• 3 examples (pict.)
• TENNLab sotware framework
  • https://www.semanticscholar.org/paper/The-TENNLab-Exploratory-Neuromorphic-Computing-Plank-Schuman/95a139d59d7551128e89b62b9d114ff1c8a27c09
  • http://neuromorphic.eecs.utk.edu/publications/2018-08-17-the-tennlab-exploratory-neuromorphic-computing-framework-submission/
• Types of Neuromorphic Implementations
  • DANNA 2- fuly digital https://dl.acm.org/citation.cfm?id=3229894
  • mr DANNA - mixed analog-digital
  • SOEN - optoelectronic

VLSI Research for Neuromorphic Computing in IBM Research

• spiking/non-spiking neural network, digital/analog implementation
  • spiking/digital - TrueNorth
  • non-spiking/digital - GPGPU, FPGA, FPU arrays
  • spiking/analog - Spiking neural network chips w/ non-volatile memory arrays
  • non-spiking/analog -
• A Scalable Multi-TeraOPS Deep Learning Processor Core for AI Training and Inference - https://ieeexplore.ieee.org/document/8502276, https://xpressdrivein.org/glo16/pdf/C04-2.PDF
  • approximate computing by reduced precision computations
• A million spiking-neuron integrated circuit with a scalable communication network and interface - http://science.sciencemag.org/content/345/6197/668 http://paulmerolla.com/merolla_main_som.pdf
  • 1M neurons, 256M synapses tileable 2D-onchip
• NVM synaptic array - eg. phase change memory (pict.)
• Analog multiply accumulation with non-volatile memory array
• NVM Weight Variation Imact on Analog Spiking Neural Network Chip - https://link.springer.com/chapter/10.1007/978-3-030-04239-4_61
• Lightweight Refresh Method for PCM-based Neuromorphic Circuits - https://www.semanticscholar.org/paper/Lightweight-Refresh-Method-for-PCM-based-Circuits-Ito-Ishii/5ae6ddfcf1820a7c1289d97197c09b598bc0724a

AI on the Edge: Frontiers for Energy-Efficient Hardware Architectures

• "Structure" is a key
• Binary/Ternary DNN accelerator VLSI 2017
  • Binary/Ternary, Reconfigurable in Memory
  • cf. https://www.researchgate.net/publication/321930684_BRein_Memory_A_Single-Chip_BinaryTernary_Reconfigurable_in-Memory_Deep_Neural_Network_Accelerator_Achieving_14_TOPS_at_06_W
• Log-Quantized DNN accelerator with 3D SRAMs
  • QUEST(Log QUantization, MIMD Parallel Engine, Die-STacking with SRAMs)
  • cf. Convolutional Neural Networks using Logarithmic Data Representation - https://arxiv.org/pdf/1603.01025.pdf
• Dynamically reconfigurable processor with AI-MAC engine
  • DRP(96-core, 333MHz) + AI-MAC(1024, 500MHz)
• The Era of "Intelligence at the Edge" will Begin
  • Common Key Features: Mostly static, dataflow rich, (self) evolvable
  • Procedure Oriented Computing -> Structure Oriented Computing: Reconfigurable HW, "Virtualized" Reconfigurable HW (post FPGA), Dataflow Oriented Machine (w/ reduced synth. cost)

Towards biologically plausible learning of spike-based cognition

• CT-AuGMEnt

Stochastic Computing for Brainware LSI

関連する？

• Large-Scale Neuromorphic Spiking Array Processors: A Quest to Mimic the Brain - https://www.frontiersin.org/articles/10.3389/fnins.2018.00891/full
• Adaptive motor control and learning in a spiking neural network realised on a mixed-signal neuromorphic processor - https://arxiv.org/pdf/1810.10801.pdf
• Dual Supervised Learning - https://arxiv.org/abs/1707.00415
• Implementation of a Liquid State Machine with Temporal Dynamics on a Novel Spiking Neuromorphic Architecture - https://www.osti.gov/servlets/purl/1405258