Posted: 2017-02-21
, Modified: 2017-02-21
Tags: neural nets
Adversarial examples in neural networks
main page
Statement
Neural networks can be easily fooled—ex. an adversary adding a small amount of noise can change the classification from “dog” to “cat” with high confidence. It can be fooled even by a weak adversary with just black-box access!
Related to making NN’s resistant: Have NN’s give a confidence bound.
Ideas:
- Use uncertainty quantification from statistics: Fisher information. See personal communication with Jacob.
- Use an ensemble of neural nets. Train an ensemble in parallel, vs. train together against a discriminator.
- Sleeping in NN
- Use some kind of calibration
- Active learning
- Make Lipschitz/other regularization.
- Boosting
Literature
- [SZSB14] Intriguing properties of neural networks paper
- Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
- Papernot 2016
- [LCLS17] Delving into Transferable Adversarial Examples and Black-box Attacks
Experiments
- Reproduce adversarial examples result on a simple dataset, e.g. MNIST.
- Try training an ensemble of NN in parallel and compare to predictions of a single one.
Theory
- Think in terms of learning theory, VC dimension…
Diversity in ML
Literature
- Putting Bayes to Sleep
- [KPRV17] Overcoming catastrophic forgetting in neural networks
GANs
The original formulation of GANs is plagued by many mathematical problems. What are mathematically better alternatives?
- Wasserstein GAN
- Neural net GAN
Decision theory and logical induction
See page on decision/game theory.
- UDT
- Reflective oracles
- Background on logic and provability
- Logical induction: the daemon (rocket) problem <> sleeping?
POMDP
Anchor POMDPs.
What are real-life problems involving POMDPs?
Alexa
- Entity extraction and feeding into knowledge graphs
- Structured knowledge graphs?
- Recognize simple queries in a reflective manner - maybe building on AIML
- Formalize modes of conversation
- Look at semantic parsing literature
- What kind of experiments could I run?
- Look for examples of conversations on popular topics. Try to engage in them.
- Do a web search with heuristics.
Logic and ML
???
What is good research?
Blogging
- Diversity series
- Boosting, etc.
- Logical induction
Learning
- Boosting, etc.
- RL
- TF in Haskell!