CS166
Chris Pollett
Nov 28, 2012
On Monday, we described how Trudy if she got lucky could use a buffer overflow to authenticate herself. Today, we begin by trying to understand how Trudy can increase her odds. To do that we need to know a little about how memory in a computer process is typically organized.
void func(int a, int b){ char buffer[10]; } void main(){ func(1, 2); }
Exercise 11.6. Recall that a canary is a special value that is pushed onto the stack after the return address.
(a) How is a canary used to prevent stack smashing attacks?
Answer. A canary is used together with run time stack checking. The run time stack checking system is invoked when the return address is popped off the stack. The system checks that before the return address a "canary" byte string still exists. This byte string often contains characters like \0 to prevent a string from overrunning the stack data. If it doesn't exist an error state is gone into.
(b) How was Microsoft's implementation of this technique, the /GS compiler option flawed?
Answer. With the /GS flag, the error state that was gone into by the run time stack checking system jumps to a user defined handler function. This handler function turned out to be specifiable by an attacker. Thus, the attacker could jump to their code.