Reading and Understanding Related Work

Step 1: Critical read, and annotation strategy

The first time you read the paper, read the abstract, then the introduction, then the conclusion. Based on these points, using a sticky note, indicate in bullet point form, what you think the paper is about, and what arguments the authors are making. Attach this sticky note to the front of your paper.
Next, read the paper from front to back. On the first pass, try to understand what the authors are saying -- understand what they did, and pick out arguments about why they did what they did, and also what they are claiming about their approach (why it's good, etc.). Also look for details on their experiment or evaluation, and think about whether you think that approach/experiment is good based on what you know.
Now that you know what is in the paper, go back to your first sticky note and check whether the authors actually did what they said they would do. Did they actually address everything they said they would address?
Read through the paper now from front to back again. Identify each argument the authors are making, and make a decision as to whether they are actually doing a good job of supporting each of their arguments. In this pass, choose one of the following frameworks, and identify each of the following points in the framework:

Framework 1

Framework 2

Problem: What is the problem the authors are addressing in their paper? What is their motivation for tackling this problem (i.e. who will care)?
Approach: What is the approach the authors are taking in this paper to address the problem? That is, if they have a solution what is the solution?
Evaluation: How do they authors evaluate their approach? How do they know it is a good one?
Contribution: What do the authors contribute to the body of knowledge/research that already exists?

Question: What is the single (big) research question the authors are trying to address in this work? Why do they care about this question?
Approach: How do the authors address this question? How do they operationalize this question within the framework of their research? If they conducted an experiment/study to address this question, what was it?
Findings: Whare the main findings that the authors discover/identify from their research?
Contribution: How do the authors' findings address the original question? What more do we know that we didn't earlier? Further, what questions does their findings raise within the broader research frame?

Consider carefully why we chose this paper. Identify the following:

How does it relate to the topic that you have chosen for your work?
What mistakes did the authors make that we can learn from?
What major findings did the authors make that we can leverage for the purpose of our work?

You can structure your critique in the following way:

Citation. Provide the full citation for the paper.
Summarize. For 4-6 sentences, summarize the paper using the framework that you used in Step 1.
Rationale. What is the relationship between this work and your _____? Why did you choose this work?
Strengths. In bullet point form, identify the strengths of the paper, and also why it is a strength (is it how they did the work, is it unusually novel, etc.). Articulate here arguments that the authors make that they really do a good job backing up.
Weaknesses. In bullet point form, identify the weaknesses of the paper, and why it is a weakness. What identify why (from a high level) it is a weakness, and also suggest an alternate approach.
Questions for author. This last section, you should imagine that you have the opportunity to talk to the original authors. If this were the case, what would you ask them about their work? (e.g. "Why did you do X instead of Y?")
What can we learn? Given that you've identified strengths and weaknesses, and also talked about how this work relates to your own now, identify what you can do to build on this work in your own. What lessons can you take? What can you do differently, etc.?