Homework: Homework is
due typewritten (including source code)
by 11:59pm on the due date.
Each assigned problem requires a solution
and an explanation (or work) detailing
how you arrived at your solution.
Cite any outside sources used to solve a problem.
When grading an assignment,
I may ask for additional information.
Note that a subset of the assigned problems
will be graded.
Zip your homework into a file named hmk.zip.
Email your work to sjsu.cs286@gmail.com.
The subject line of your email must be of the form:
CS286HMK assignmentnumber yourlastname last4digitofyourstudentnumber
That is, the subject line must consist of four identifiers.
There is no space within an identifier and each identifier
is separated by a space.
- Assignment 1: Due Tuesday, September 1
Obtain and install OllyDbg (or a similar tool). Download the
file serial.zip
and extract the files serial.exe and serial2.exe.
For each program, create a patched version for which any serial number
works, and also determine the serial number.
- Assignment 2: Due Thursday, September 10
- Write a C program that includes all of the elements listed below.
Then compile your program and disassemble the resulting exe.
Using only a static view of the
disassembled code, determine the assembly code that corresponds to each
element. Your solution must clearly show each element in source
code and the corresponding disassembly.
- simple if (i.e., no else clause)
- "shorthand" if, i.e., use the ?
operator---for example, x = (y < 0) ? -y : y;
- compound if (i.e., include && or || or both)
- if/else
- if/elseif/elseif/.../else
- switch statement (with at least 3 case statements
and a default case)
- simple for loop
- for loop with a break
- for loop with a continue
- while loop
- do loop
- array
- function call (include both the CALL and RET)
- each of the integer arithmetic operations,
+, -, *, /, and %
- each of the floating point arithmetic operations,
+, -, *, and /
- For serial2.exe (see assignment 1), analyze the method
used to obfuscate the strings. Give another method for obfuscating
literals that you believe would be more difficult to reverse.
- Assignment 3: Due Tuesday, September 15
- Examine the PE file ntdll.dll (in the system32 directory) and
provide the following information. Give all numbers in hex
unless otherwise specified.
- Number of sections and name of each section
- Size of the optional header.
- "Magic number" (in hex and in ASCII).
- Size of code.
- Size of initialized data.
- Address of entry point.
- Base of code.
- Base of data.
- Number of exported "symbols" (in decimal).
- Name and entry point of export symbol with ordinal 27.
- Size of relocation table.
- Examine the PE file crypt32.dll (in the system32 directory) and
provide the following information. Give all numbers in hex
unless otherwise specified.
- Number of sections and name of each section
- Size of the optional header.
- "Magic number" (in hex and in ASCII).
- Size of code.
- Size of initialized data.
- Address of entry point.
- Base of code.
- Base of data.
- Number of exported "symbols" (in decimal).
- Name and entry point of export symbol with ordinal 1027.
- Size of relocation table.
- Number of DLLs that are imported.
- RVA of imported symbol "CryptGenRandom"
- Assignment 4: Due Thursday, September 24
- Write a Java application and obfuscate it using ProGuard. Only use
options under the "obfuscation" button (i.e., no shrinking, no optimization, etc.)
and select the "print mapping" option.
Give your obfuscated program to someone else in the class, who
must patch the program. Give your victim precise instructions on what
the patched program is supposed to do. Be reasonable!
- You must also obtain an obfuscated program (created using ProGuard)
from another student and patch it according to the other student's
instructions. You cannot obtain this program from the same person who
will be patching your program.
- Obfuscate the Java application you wrote for Problem 1 using SandMark.
Select the "obfuscate" tab and the "Class Encrypter" option. You may also select other
obfuscation options if desired. The challenge is to generate a decompiled version of your
program directly from the obfuscated code (i.e., without using SandMark
or any similar tool and without reference to the original Java code).
This will require that you decrypt the encrypted classes in the obfuscated code.
- Assignment 5: Due Tuesday, October 6, before 3:00pm
Note: Email this assignment to stamp@cs.sjsu.edu
with subject line "CS286 Project 1"
- Turn in a Windows exe and the source code for your Project 1 software,
along with detailed instructions on how to compile it.
- Provide a written description of what your software does
and what you would expect another student team to do with your project
to break its security. You must be specific when discussing the
security feature you expect to be broken. Note that this description will
be given to the student team that is assigned to your project.
- Provide a detailed description of all anti-reversing
techniques that you applied to your software. Include a discussion of the
effectiveness of each technique with respect to your particular software.
This information will not be given to the student team that is assigned
to your project.
- Assignment 6: Due Tuesday, October 13
- Modify the program
serial.c
to test for a debugger using IsDebuggerPresent.
The program should silently terminate if a debugger is detected.
Turn in your source code along with an explanation of what happens when you
try to debug the code.
- Turn in a screenshot showing that you were able to debug your
program in problem 1, in spite of the IsDebuggerPresent code. Write
a short explanation of how you were able to bypass the IsDebuggerPresent check.
- Repeat problem 1, but instead of using IsDebuggerPresent, use the
assembly code that appears on slide 39 of the
SRE_anti-reversing.ppt
slides. How much more difficult would it be to bypass this debugger check?
- Using serial.c, test each of the anti-disassembly techniques that appear on the
following SRE_anti-reversing.ppt slide: 45, 47, 48, and 49. In each case, turn in your source
code and describe the resulting disassembly that is produced in OllyDbg and in IDAPro.
- Again using serial.c, develop your own anti-disassembly technique that
does a better job of confusing both OllyDbg and IDAPro than
any of the techniques tested in the previous problem. Turn in your source
code and explain what you did and describe how much the assembly code
was garbled in both OllyDbg and IDAPro. How difficult would it be to
reverse engineer code that included your anti-disassembly technique?
- Assignment 7: Due Thursday, October 22
- "Rip" the keygen algorithm from the code, keygen.exe, found
here.
That is, extract the keygen assembly code from the application and
use it directly in your own program, where your program takes any
valid username as input and produces a valid serial number.
- Analyze the keygen algorithm used in keygen.exe and provide
a description of the process used.
- Which is easier, ripping the keygen algorithm from the
disassembly of keygen.exe, or analyzing the code and writing
an equivalent version from scratch? Explain.
- Assignment 8: Due
Thursday, November 12
Tuesday, November 17
The program "Defender.exe" can be found
here
and the slides that cover this application are
here.
Write your own application, "defender2" that requires a username and a serial number
as input. Your program must implement a keygen algorithm (of your own design)
and check that the username and serial number are valid.
- Your program, defender2, must include the following
security features found in Defender.exe.
- Encrypted code. The encrypted code should only be decrypted when
it is reached during normal execution, as opposed to being decrypted when
the application first begins.
- A secondary "killer" thread that kills the application if the main thread is
too slow (as would be the case if the application is being debugged).
- Show that you can defeat both of the protections in you application, defender2.
- Assignment 9: Due TBD
- Assignment 10: Due TBD
- Assignment 11: Due TBD