Week 1 Research Skills:

Introduction to Computer
Science + Electrical Engineering
Research
What is research?
● Systematically studying a topic to expand the current
knowledge/understanding within a certain field.
○ Document!
○ Research should be objective, reliable, and reproducible.

● Using predictions to attempt to come up with potential solutions to a

particular problem or question of interest.
○ Identifying a gap from other research
○ Observing a certain problem, then trying to derive a way to address it
 Who does research in computer science + electrical
engineering?
Academic Scientists Professors Industry Professionals Physicians

High school students like you!

Undergraduates, Graduate students, Post-doctoral scholars
Open-source software developers
Translational scholars

Virtually anyone (with access to necessary resources)

 Why do we have Research Skills lectures?
● sci-MI wants to improve youth accessibility to the world of science research,
especially those from historically underrepresented communities in science.

● Research in computer science and electrical engineering in particular can

typically be done from anywhere, not just inside of a university.

All icons made by Freepik from www.flaticon.com

What do we cover in the Research Skills lectures?
Week X: How do we conduct research ethically (Ethics Bootcamp)

Week 1: What is research in CS + EE and how do we do it

Week 2: How to read, find, and use scientific literature

Week 3: What is open-source software and how can we use it for research

Week 4: How to present data and use statistical testing for CS + EE

Week 5: What careers and next steps exist for CS + EE researchers

Week 6: Symposium!
Why are CSMP and EEMP sharing Research
Skills Lectures?
● The distinction between what is CS and what is EE research is very
university-, subfield-, and department-dependent!
● Many of the research skills we’re going to cover are equally important in CS
and EE research
● At the extreme, CS and EE research can be very different. But there are many
areas of overlap between the two fields.
Types of Research
● Experimental - designing an experiment to test a hypothesis and reach
new conclusions about a certain topic
○ For example: user studies or timing experiments
● Theoretical - using mathematical proofs to analyze and design systems
+ behaviors
● Correlational - finding correlations between different trends in data
● Observational - using surveys and ethnographies to observe population
trends
More Types of Research
● Qualitative - collecting non-numerical data to answer a research
question
○ For example: conducting interviews or analyzing text data

● Historical - using historical accounts to build a new telling of history

○ For example: track development or create a timeline

● Content/media analysis - examining the cultural content of media

○ For example: identifying trends or examination for bias
CS + EE Research tends to focus on:
Experimental: Theoretical:
Experimental research in CS + EE often 1. Fully abstract mathematical proofs -
results include fundamental limitations,
begins with an idea that is designed and
impossibility results, etc.
manufactured by the research team 2. System design - designing new algorithms,
protocols, and other mathematical systems
1. Computational - includes modeling and and proving that they work
testing the design computationally a. e.g., any algorithm, control system, or
2. Physical - includes manufacturing and communication protocol
3. Model design - designing a new way to
testing a physical device / design
model some system and proving some results
a. This tends to be more prominent in EE
about that model
research than CS a. e.g., how to share a resource such as
bandwidth dynamically with multiple users,
how to model wireless fading, etc.
CS + EE Research tends to focus on:
Experimental: Theoretical:
Experimental research in CS + EE often 1. Fully abstract mathematical proofs -
results include fundamental limitations,
begins with an idea that is designed and
manufactured by the research with engineering research inimpossibility
Asteam general, the results, etc.
2. System design - designing new algorithms,
ultimate goal of both experimental and
protocols, and other mathematical systems
1. Computational - theoretical research
includes modeling and is to be usedandforproving
somethat kindthey work
of application - sometimes the application
testing the design computationally a. is
e.g., any algorithm, control system, or
2. extremely and
Physical - includes manufacturing communication
specific, while other times, the protocol
3. Model design - designing a new way to
application can be very broad.
testing a physical device / design
model some system and proving some results
a. This tends to be more prominent in EE
about that model
research than CS a. e.g., how to share a resource such as
bandwidth dynamically with multiple users,
how to model wireless fading, etc.
 Applications of CS are everywhere!

Architect Artificial
ure Intelligen
Theory ce + Biomedic
Robotics Machine ine +
Learning Health
Computati
Reinforce onal
ment Biology +
Learning Neuroscie
nce

EE

Operating
+ Computer
Distributed Graphics
Systems Natural
Computer
Language
Human- Security
Processing Computer
Computer
Data Vision
Interaction
SCience

And more!!
Source: https://www.cs.stanford.edu/people-cs/faculty-research
 Applications of EE are everywhere!
AI,
Nanotech Machine
Computer nology, learning,
Optimizat Photonic- Energy,
architectur nanofabri Environm
e (incl. ion s
cation, ent,
efficiency materials
reliability, Sustain-a
Informat- security) bility Comput-
ion, ational
Learning, Sensing,
Communi Imaging,
cations Displays
EE

Software
Signal systems,
Processin Environm- security,
ental distributed
g+ Integrat-e
sensors, systems
Control Quantum d Circuits
embedded networks
science +
systems,
remote and Robotics Systems
engineeri Biology +
sensing
ng Health

And more!!
Source: https://ee.stanford.edu/research/the-big-picture
Questions, Comments,
or Concerns?
Common misconceptions about research
● Research isn’t just done in academic institutions
● Research isn’t just done in STEM
○ Research is an essential part of studying the social sciences and arts too!
● There is more than one correct way of conducting research
○ As long as research is conducted ethically and according to regulations, anyone
can design an experiment.
● Research results are not final!
○ While a team of researchers may have reached a conclusion based on their
results, their conclusions can be tested by other researchers to confirm their
validity.
How does a research project start?
1. Start with an observation and/or a question
○ Why does x sometimes fail to follow y?
○ How did we arrive at x in history?
○ What is the effect of x on population group y?
○ What are the ethical implications of topic or practice x?
○ This paper investigates the theory of x, can I apply this to y in practice?
○ Thing x is done in practice, can I prove that this is the optimal solution?
○ What happens if I perturb existing process x by changing y?
○ This paper establishes x on y, what happens if I apply x to b?
○ Process x is typically undertaken by hand, can we automate this?
○ How do users interact with software x? If we change y, does their behavior change?
 Breakout Session!
Share with your group which of the previous
observations/questions (or one of your own!) you would
want to explore for your symposium presentation!
How does a research project start?
2. Conduct research to get a good background on
the topic of interest
1. Read secondary literature sources to get an idea of
where the field is at (systematic and scoping reviews)
2. Refine your interest/question
3. Read primary literature on your narrowed area of focus
How does a research project start?

3. Create a hypothesis that answers your initial

question

● Method y will outperform existing method x

● Thing y can be automated by method x
● Opinions about software x differ between demographic
groups y
How does a research project start?
4. Design an experiment to test that hypothesis

● Conduct a survey
● Design and train model x, software y, or extension z
● Run user study

5. Once completed, analyze the results and report your conclusions

● Publish a paper
● Present a poster or slides at a conference/symposium/gathering
● Put a preprint/white paper on an online repository
● Tweet about it
A quick look at research papers
Most research papers have the same format:

1. Abstract - a short overview of the paper, it’s goals, results, and relevance.
2. Background/Related Work - a review of the literature
3. Materials and Methods - the actual experimental methods
4. Results - the experiments’ results
5. Discussion - a more in-depth look at the results, their meaning, and the
study’s limitations
6. Conclusion - implications and future directions
7. References - proper citations of referenced articles
 Methods sometimes
CS papers are changing! moved to after Results
Most research papers have the same format:
Results/Methods may
be removed in favor of
1. Abstract - a short overview of the paper, it’s goals, results,
more paper-specific
and relevance. content headings
2. Background/Related Work - a review of the literature
Modern CS papers
3. Materials and Methods - the actual experimental methods typically combine the
4. Results - the experiments’ results Discussion and
5. Discussion - a more in-depth look at the results, their Conclusion into a
single section
meaning, and the study’s limitations
6. Conclusion - implications and future directions Having extensive
7. References - proper citations of referenced articles supplementary
information is
8. Appendix/Supplement - online-only material providing
becoming very
additional information, proofs, experiments, etc. popular, especially in
AI
The format of typical EE papers often depends on the
field / application!
Most research papers, even if these are in a different order, have these components:

1. Abstract - a short overview of the paper, it’s goals, results, and relevance.
2. Background/Related Work - a review of the literature
3. Materials and Methods - the actual experimental methods
4. Results - the experiments’ results
5. Discussion - a more in-depth look at the results, their meaning, and the study’s
limitations
6. Conclusion - implications and future directions
7. References - proper citations of referenced articles
8. Appendix/Supplement - online-only material providing additional information,
proofs, experiments, etc.
Questions, Comments,
or Concerns?
Developing a Hypothesis
● A hypothesis is an educated guess of what the researcher predicts will be the
outcome of the study.
○ The hypothesis should be declared before the actual experiment takes place
○ It should be precise and testable
○ This type of hypothesis is more common in experimental forms of CS research, such as when
doing A/B testing for human-centered design

● By the end of an experiment, we cannot say whether a hypothesis is “right or

wrong”
CS + EE research frequently does not explicitly state the
hypothesis. It can be a good exercise to try and
reconstruct the authors’ hypotheses from the paper text.

● Hypothesis: our model design A will outperform model design B on

certain benchmarks
● Hypothesis: a software tool to perform task A can be constructed
Not all CS + EE research
is hypothesis-driven!
Research can be driven by an observed problem or
inefficiency, a lack of theoretical/ethical framework,
or some other reason!
Theory vs. Hypothesis
Hypothesis

A hypothesis is a tentative explanation about an observation that can be tested.

Any one observation usually comes with an array of hypotheses. If you observe that a
swan is white, your hypothesis could be that it's painted, or it was bleached by the sun, or
its feathers just lack pigment. You can then investigate all of those hypotheses and come
away with the one that's most supported by the evidence, if any.

https://www.discovery.com/science/Difference-Between-Fact-Hypothesis-Theory-Law-Science
Theory vs. Hypothesis
Theory

A theory is an explanation of some aspect of the natural world that's well-substantiated

by facts, tested hypotheses, and laws. A theory is the granddaddy of all scientific
statements, which is why it makes no sense to say that evolution is "just a theory."

The strength of a scientific theory is that it's built on a sturdy enough foundation that even if
you find a few cracks in it, you can trust that the structure as a whole will remain standing.

https://www.discovery.com/science/Difference-Between-Fact-Hypothesis-Theory-Law-Science
Computational Complexity Theory is an example
of a theory in computer science + electrical
engineering!

https://www.bigocheatsheet.com/
Nyquist–Shannon sampling theorem is an
example of a theory in electrical engineering!

https://briansradiologylearningdiary.wordpress.com/2017/08/29/ch-3-7-nyquist-frequency/
The Lottery Ticket Hypothesis is an example of a
hypothesis in computer science!

This recently-proposed hypothesis (2019) is currently undergoing

experimental validation in the scientific community. It hasn’t yet reached
the widespread acceptance needed to be established as a theory.
How many experiments is “enough”?
It depends! There is a whole field of statistics called sample size determination which
works on this problem!

● Experiments must be reliable, producing consistent results each time you run them
under the same conditions.

● Experiments should be valid,

showing that you are truly
measuring what you claim to be
measuring.
Neither reliable
Reliable, but not valid Valid and reliable!
nor valid
https://opentextbooks.uregina.ca/foundationsofscoialworkresearch/chapter/5-4-measurement-quality/
In other words, experiments must be
reproducible, leading to consistent and
accurate results and analysis.

Even once the data is part of your science

textbook, it can still be updated!
Questions, Comments,
or Concerns?
Documenting the data you collect in research
● Take in-depth notes, use images, or create your own diagram(s)
○ Data can be used to create tables or figures
○ Create backups of everything!
○ When in doubt, write it down
○ Maybe keep a lab “notebook”
○ Symposium: make at least one original table and figure to represent your data
collection

● Explain how the data helps you test your hypothesis

○ How does it suggest a potential solution to your research question?
○ You should have a reason for every piece of data you collect
 Articles are a
type of
documentation, What kinds of articles are
there?
designed to
share scientific
developments.
Primary Research Articles
● The authors conducted their own original research
○ Empirical: based on observations and/or experience
○ Theoretical: mathematical or ethical analysis, may or may not contain experiments
● Typically found in peer-reviewed journals or at conferences
○ Evaluated by other experts in the field
○ Ensures academic scientific quality by checking for validity, errors, biases, etc.
○

Secondary research articles summarize and/or evaluate data from

primary research articles.
Review Articles
● Provide summaries on a previously researched topic
● Can come up with wider implications by putting together information
gathered from different sources
● Instead of recruiting participants for a new study, secondary research articles
have selection criteria for what articles they will use.

Types of reviews: scoping, systematic, surveys, mixed methods, rapid,

state-of-the-art, mapping (https://library.shu.edu/ReviewArticles/Types)
Systematic Review
● Very comprehensive and precise paper attempting to answer specific scientific
questions
○ Systematic reviews search multiple databases to answer a pre-defined question
○ Uses several criteria to consider accuracy and adequacy of many articles
○ Methods for finding and analyzing sources are meant to be replicated
○ Articles are chosen to be included based on, ideally, non-subjective criteria
Systematic Review
● It combines primary information extracted from eligible studies
○ These studies might be research articles, theses/dissertations, patents, or
preprints
○ This information is used to develop broader conclusions and hypotheses while
recognizing the limitations of the studies considered
Meta-Analyses
● The statistical process of analyzing and combining results from
several similar studies
○ Uses statistical methods to summarize results of studies and derive new insights
○ Can be included as part of a systematic review, but not all systematic reviews
have meta-analyses
Meta-Analyses
● Should only be used with data that is “similar enough”
○ Bad methodology or trying to combine data that is too broad can be misleading
○ Need to make sure that the studies used are trying to answer the same question
○ Typically need to consider publication bias
Other kinds of secondary research articles
● Case studies (looking at a unique scenario or illustrative example)
● Clinical trials/studies
● Working group meeting findings
● Technical/Expert opinions
● Perspectives (future directions)
● Comments
Questions, Comments,
or Concerns?
How to Read a Primary Research Article
Week 1 Journal Club Sneak Peek

● Unpack your goal in reading the paper

● Think about what questions the author(s) intended to answer and their attempts
to do so
● What are the key findings? How are they highlighted through visual elements?
● How can the research improve (plausibly and ethically)?
● Think about future directions for this area of research
● Ask questions about the paper and for tips on understanding it better!

For more information and tips, check out:

Ten Simple Rules for Reading a Scientific Paper (Carey et al., 2020)
(Week 1 Journal Club reading!)
Reading research articles takes time and
practice!
Research in Computer Science + Electrical
Engineering
● CS + EE research can intersect with
basically any other field of study!
○ There is a huge need for computational
researchers to support other fields
○ CS + EE also have many different subfields

● New discoveries are being made every day!

● We need to continue working to overcome
societal barriers in research
Source: https://info.arxiv.org/help/stats/2021_by_area/index.html#cs_yearly
Questions, Comments,
or Concerns?