Neural Net Captcha Cracker

Geetika Garg, Chris Pollett (presenting)

San Jose State University

Future Technology Conference, San Francisco, Dec 6, 2016


Today, I'd like to report on work of myself and Geetika Garg on training neural networks to break image-based captchas.



Example Captcha Image 1 Example Captcha Image 2 Example Captcha Image 3

Prior Work

What does it mean to break a Captcha?

Artificial Neurons

Neural Nets

Our Networks

Inspiration for Our Networks

Our Test Set


Experiments - Individual Characters

Type of modelIndividual Character Accuracy
LSTM fixed length (simple dataset)99.9%
LSTM fixed length (complex dataset)98.48%
Multiple Softmax fixed length (simple dataset)99.8%
Multiple Softmax fixed length (complex dataset)98.96%
LSTM variable length with fixed length data99.5%
LSTM variable length with variable length data97.31%

Experiments - Sequence Correctness

Type of modelSequency Accuracy
LSTM fixed length (simple dataset)99.8%
LSTM fixed length (complex dataset)91%
Multiple Softmax fixed length (simple dataset)99%
Multiple Softmax fixed length (complex dataset)96%
LSTM variable length with fixed length data98%
LSTM variable length with variable length data81%

Experiments - Versus Humans



(von Anh, et al 2003)
L. von Ahn, M. Blum. N. J. Hopper, and J. Langford. CAPTCHA: Using Hard AI Problems for Security. EUROCRYPT 2003: International Conference on the Theory and Applications of Cryptographic Techniques. 2003. pp. 294--311.
(Chellapilla, et al 2004)
K. Chellapilla and P. Y. Simard. Using Machine Learning to Break Visual Human Interaction Proofs (HIPs). Advances in Neural Information Processing Systems. Volume 17. NIPS 2004. pp. 265--272. 2004.
(Goodfellow, et al 2014)
I. J. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud, V. Shet. Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks. International Conference on Learning Representations 2014. arXiv:1312.6082. 2014.
(Haykin 2011)
S. O. Haykin. Neural Networks and Learning Machines. 3rd Ed. Pearson. 2011.
(LeCun, et al 1989)
Y. LeCun, B. Boser, J. S. Denke, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel. Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems. Volume 2. NIPS 1989. pp. 396--404. 1989.
(McCulloch and Pitts 1943)
S. McCulloch and W. H. Pitts. Resource Description for A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics. Volume 5. pp. 115--133. 1943.
(Mori and Malik 2003)
G. Mori and J. Malik. Recognizing Objects in Adversarial Clutter: Breaking a Visual CAPTCHA. IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2003. Vol. 1. IEEE Computer Society. pp.134--141. 2003.
(Naor 1996)
M. Naor. Verification of a Human in the Loop, or Identification via the Turing Test. Unpublished Manuscript. 1996.
(Netzer, et al 2011)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng Reading Digits in Natural Images with Unsupervised Feature Learning NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011.
(Vinyals, et al 2015)
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and Tell: A Neural Image Caption Generator. IEEE Conference on Computer Vision and Pattern Recognition. pp. 3156--3164. 2015.