You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A lot of the best research in machine learning comes from collaborations.
112
+
In fact, many of the most significant papers in the last few years (TensorFlow, AlphaGo, etc) come from collaborations of 20+ people.
113
+
These collaborations are made possible by goodwill and trust between researchers.
114
+
</p>
108
115
109
-
<p>A lot of the best research in machine learning comes from collaborations. In fact, many of the most significant papers in the last few years (TensorFlow, AlphaGo, etc) come from collaborations of 20+ people.</p>
116
+
<p>
117
+
This goodwill and trust is a precious shared resource, and it can be a fragile thing.
118
+
When people work together, it’s easy to have conflict, especially around attributon and credit.
119
+
If dealt with poorly, attribution issues can fester.
120
+
I'm aware of several cases of collaborations dieing, or people leaving teams and organizations,
121
+
where and the underlying issue was hurt feelings and lost trust around collaboration.
122
+
This strikes me as rather sad.
123
+
</p>
110
124
111
-
<p>When people work together, it’s easy to have conflict -- especially around credit. This can easily turn into hurt feelings and damage collaborations. To avoid this, it’s not enough to avoid academic misappropriation or not giving credit. We should aim to avoid any <i>feeling or appearance</i> of misappropriation as well, if we want to foster a healthy environment for collaborations.</p>
112
125
113
-
<p>It’s especially important for us to be mindful of credit issues, because they can easily be perceived as, and likely often are, linked to privilege or power gradients. This might be gender or race, but it can also be things like being a remote collaborator (ie. geographically removed from others), being an engineer or designer instead of a researcher, or being at a lower level professionally. A perception that diverse or junior collaborators are taken advantage of is deeply toxic.</p>
114
126
115
-
<p>The theory of this document is that it’s easier to agree upon general principles, independent of particular situations. Then, when a particular problem arises, you can talk about it in the context of these more general principles.</p>
127
+
<!-- How can we add to the resvoir of trust? -->
128
+
<p>
129
+
We often talk about credit issues in kind of binary terms.
130
+
But if the thing we care about is this shared trust, I think it's not enough to avoid doing anything "wrong".
131
+
We must also avoid any <i>feeling or appearance</i> of unfairness.
132
+
In fact, we'd ideally actively cultivate the opposite -- behave in ways that add back to the pool of goodwill.
133
+
</p>
116
134
117
-
<p>This document reflects my personal working principles. This is a hard problem, and I don’t think these are a perfect solution. I expect them to improve over time. I’d love your comments!</p>
135
+
<p>
136
+
(We should also be mindful that credit issues can easily be perceived as, and likely often are, linked to privilege or power gradients.
137
+
This might be gender or race, but it can also be things like being a remote collaborator (ie. geographically removed from others), being an engineer or designer instead of a researcher, or being at a lower level professionally.
138
+
A perception that junior collaborators or those from under-represented minorities are taken advantage not only harms the research community, but the larger cause of making sure all humans are treated fairly.)
139
+
</p>
140
+
141
+
<p>
142
+
This is a hard problem, and one that lots of people have thought about before.
143
+
I don't have any perfect anwers, but I'd like to share some personal working principles that I feel like have been helpful for me.
144
+
I’d love your comments and feedback!
145
+
</p>
118
146
119
147
<h2>Core Principles</h2>
120
-
148
+
121
149
<style>
122
150
ulli {
123
151
margin-top:12px;
124
152
}
153
+
154
+
.tight-listli {
155
+
margin-top:6px;
156
+
}
125
157
</style>
126
158
127
159
<ul>
128
-
160
+
129
161
<li><b>Always check in with any person who could plausibly be an author</b> or feel like they should be, <i>even if you disagree</i>. Never have authorship or authorship order be decided behind closed doors or without giving people an opportunity to advocate for themselves.</li>
130
162
131
163
<li><b>Err on the side of sharing credit.</b> Credit isn’t zero sum. It is often in everyone’s benefit to be generous with credit, because it creates an incentive for others to help in the future. It also makes sense to be risk-averse to the possibility of not crediting someone who deserves it, because the harm of not crediting someone who deserves it is often greater than the harm of crediting someone who doesn't deserve it.</li>
132
164
133
165
<li><b>Acknowledge anyone you can remember talking to</b> about your research significantly. It costs nothing and builds good will. You can still use stronger language to highlight people who helped you more.</li>
134
166
135
-
<li><b>Don’t reveal someone else’s work or merge it into your own without their consent.</b></li>
167
+
<li><b>Avoid diffusion of responsibility</b>. For example, have someone clearly responsible for checking in with everyone on authorship. </li>
168
+
169
+
<li><b>Don’t reveal someone else’s unpublished work or merge it into your own without their consent.</b></li>
136
170
137
171
<li><b>Remember that you are likely overestimating your own contributions.</b></li>
138
172
139
173
<li><b>Act in ways that will make people want to work with you.</b> Enthusiastic collaborators are one of the most precious thing you can have as a researcher.</li>
140
-
174
+
141
175
</ul>
142
176
143
177
144
178
<h2>Pre-commit yourself to treating others well.</h2>
145
-
146
-
<p>Any place you can pre-commit yourself to treating others well is an opportunity. It protects you against accidentally hurting someone and also against others worrying that you might hurt them.</p>
147
-
179
+
180
+
<p>
181
+
Most attribution issues are mistakes rather than malice.
182
+
Adopting good collaboration practices is our best defense against making hurtful errors!
183
+
</p>
184
+
185
+
<p>
186
+
More fundamentally, any place you can pre-commit yourself to treating others well is an opportunity to invest in trust and goodwill.
187
+
If you make ensuring that others are treated well a priority, those around you will notice.
188
+
It will foster emotional safety within your working group,
189
+
and it will make others want to follow your example.
190
+
</p>
191
+
148
192
<ul>
149
193
150
194
<li><b>Mention collaborators when sending emails about your work or speaking publicly.</b> This is especially important if you are communicating with leadership or PR.</li>
151
195
152
196
<li><b>In drafts, put “author order not finalized” under author list</b>, use a collective authorship as a placeholder, don’t list authors, etc. This avoids an appearance that you’ve unilaterally determined the authorship order and creates a natural reminder to have a conversation about authorship.</li>
153
197
154
198
<li><b>In drafts, keep a running list of people to acknowledge.</b> This reduces the risk of you forgetting to acknowledge someone. It also signals to them that you’re taking this stuff seriously.</li>
155
-
199
+
200
+
<li><b>List collaborators in working notes.</b>
201
+
If you refer back to them in the future, this will allow you to know who you were talking to and getting ideas from at the time.
202
+
</li>
203
+
156
204
</ul>
157
-
205
+
158
206
159
207
<h2>Author Contributions / Order</h2>
160
-
161
-
<p>Authorship order can be a tricky topic. It often helps to make credit distribution more nuanced, flexible, and continuous.</p>
208
+
209
+
<p>
210
+
Authorship order can be a tricky topic.
211
+
It often helps to make credit distribution more nuanced, flexible, and continuous.
212
+
</p>
162
213
163
214
<ul>
164
-
165
-
<li><b>Consider an “Author Contributions Statement”</b> in multi-author papers saying what each author did. This is standard practice in many academic fields and required by many academic journals (eg. Nature, PLoS). Often, authors make contributions of different kinds and different magnitudes. A contributions statement can distribute credit much more fairly than authorship order. It allows much more flexible and continuous distributions of credit, which often makes things less contentious. It can also reduce “honorary authorship” problems, where senior people are on papers they didn’t contribute to, by surfacing their level of involvement.</li>
215
+
216
+
<li><b>Consider an “Author Contributions Statement”</b> in multi-author papers saying what each author did.
217
+
This is standard practice in many academic fields and required by many academic journals (eg. Nature, PLoS).
218
+
Often, authors make contributions of different kinds and different magnitudes.
219
+
A contributions statement can capture this nuance much clearly than authorship order, allowing for fairer credit distribution.
220
+
In my experience, this flexible and continuous nature can also makes discussing credit less contentious.
221
+
Finally, contributions statements may also reduce “honorary authorship” problems, where senior people are on papers they didn’t contribute to, by surfacing their level of involvement.</li>
166
222
167
223
<!--<li>Writing an author contributions statement may sound contentious, if you imagine having </li> -->
168
224
169
-
<li><b>Write a contributions statement everyone agrees with before discussing author order.</b> Writing a contributions statement is often easier and more cooperative than discussing authorship order, because it’s more flexible, more objective, and in a strange way positive sum. (It’s also less zero sum in a strange way, because people are often attached to getting credit for particular things they did that feel important to them, and because listing all the work makes it feel like there’s more credit to go around.) It helps build empathy for the contributions others made, reduces the stakes of authorship order (because it matters less with a contributions statement), and gives a common starting point for a conversation.</li>
225
+
<li><b>Write a contributions statement everyone agrees with before discussing author order.</b> Writing a contributions statement is often easier and more cooperative than discussing authorship order, because it’s more flexible, more objective, and in a strange way positive sum. (It’s also less zero sum in a strange way, because people are often attached to getting credit for particular things they did that feel important to them, and because listing all the work makes it feel like there’s more credit to go around.) It helps build empathy for the contributions others made, reduces the stakes of authorship order (because it matters less with a contributions statement), and gives a common starting point for a conversation.</li>
226
+
227
+
<li><b>Understand people’s underlying motivations and needs.</b> If people disagree on authorship, it may help to understand what each person actually cares about. Sometimes this isn’t actually a particular author position, but making sure that they get credit within an organization, or with someone they really respect. Sometimes there’s a pragmatic reason why someone really needs a particular position.</li>
170
228
171
-
<li><b>Understand people’s underlying motivations and needs.</b> If people disagree on authorship, it may help to understand what each person actually cares about. Sometimes this isn’t actually a particular author position, but making sure that they get credit within an organization, or with someone they really respect. Sometimes there’s a pragmatic reason why someone really needs a particular position.</li>
229
+
<li><b>If there are multiple “primary” authors consider joint first authorship, randomization, or both.</b></li>
172
230
173
-
<li><b>If there are multiple “primary” authors consider joint first authorship, randomization, or both.</b></li>
231
+
<li><b>If you can’t find an order all authors feel good about, consider a mediated conversation or escalation.</b></li>
174
232
175
-
<li><b>If you can’t find an order all authors feel good about, consider a mediated conversation or escalation.</b></li>
176
-
177
233
</ul>
178
234
179
235
180
236
<h2>Mediation / Escalation</h2>
181
237
238
+
<p>
239
+
Ideally, we'd avoid all attribution conflicts with communication, thoughtfullness, and good collaboration practices.
240
+
But we're not perfect, and we live in the real world, so conflicts do happen.
241
+
Bringing someone else in to mediate the conflict is often the best tool.
242
+
</p>
243
+
182
244
<ul>
183
-
<li><b>In cases of conflict, involve a neutral third party.</b> This is ideally the Lowest Common Manager, or independent respected researchers who are willing to give advice or facilitate conversations on attribution situations.</li>
245
+
<li><b>In cases of conflict, involve a neutral third party.</b>
246
+
This is ideally the Lowest Common Manager, or independent respected researchers who are willing to give advice or facilitate conversations on attribution situations.</li>
247
+
248
+
<li><b>It’s essential for the third party to be neutral.</b>
249
+
A perception that the conflict was adjudicated by a biased party can make things worse.</li>
250
+
251
+
<li><b>Managers and senior researchers need to be compasionate, non-judgemental, and take attribution issues seriously.</b>
252
+
People are often ashamed to bring up credit issues, because they worry they're being unreasonable or that they'll be perceived as such.
253
+
This is true even when they're feeling hurt and alienated.
254
+
</li>
184
255
185
-
<li><b>It’s essential for the third party to be neutral.</b> A perception that the conflict was adjudicated by a biased party can make things worse.</li>
186
256
</ul>
187
257
258
+
188
259
<h2>Situations to Look Out For</h2>
189
-
190
-
Be especially careful if any of the following risk factors are present:
191
260
192
-
<ul>
261
+
<p>Be especially careful if any of the following risk factors are present:</p>
262
+
263
+
<ulclass="tight-list">
193
264
<li>Borrowing or building on unpublished work.</li>
194
265
<li>Collaborators involved in early parts of a project or closely related projects, but not the published result.</li>
195
266
<li>Loss/destruction of code history (eg. submitting code written by someone else).</li>
@@ -202,24 +273,35 @@ <h2>Situations to Look Out For</h2>
202
273
203
274
204
275
<h2>Citing others</h2>
205
-
206
-
Citation serves an important dual role: helping readers find related work, and allocating credit within the research community. This second role makes it important to handle well: it can genuinely affect people’s careers.
276
+
277
+
<p>
278
+
Citation serves an important dual role: helping readers find related work, and allocating credit within the research community.
279
+
This second role makes it especially important to handle well: it can genuinely affect people’s careers.
280
+
In some cases, it may even effect their lives in more profound ways such as immigration status.
281
+
</p>
207
282
208
283
<ul>
209
284
<li><b>Cite all work you significantly build on.</b> This includes infrastructure, such as Theano, Pytorch, TensorFlow, R, or Genome Analysis Toolkit. It can also involve data such as ImageNet.</li>
210
285
211
286
<li><b>Citation is still necessary when there isn’t a “paper version” of something.</b> If they helped or influenced your work, cite blog posts, artifacts such as code, private correspondence, or unpublished work. (This is what BibTeX <code>@misc</code> is for.)</li>
212
287
213
-
<li><b>Citation is especially important if work isn't highly cited or the author is junior.</b></li>
288
+
<li><b>Citation is especially important if work isn't highly cited or the author is junior.</b>
289
+
</li>
214
290
215
291
<li><b>Getting citation right is especially important in review papers.</b></li>
216
292
</ul>
217
293
218
294
<h2>Acknowledgments</h2>
219
295
220
-
<p>This document greatly benefited from conversations with Michael Page, Catherine Olsson, and Michael Nielsen. I’m also grateful to the comments of Shan Carter, Ian Johnson, Dario Amodei, Holden Karnofsky, Smitha Milli, Nick Beckstead, Arvind Satyanarayan, Yomna Nasser, Matt Hoffman, Emma Pierson, Martin Abadi, Greg Corrado, Amy McDonald, Jeff Dean, Samy Bengio, Sageev Oore, Konstantinos Bousmalis, Peter Liu, Andrew Dai, Jasper Snoek, and Vincent Vanhoucke.</p>
296
+
<p>
297
+
This document greatly benefited from conversations with Michael Page, Catherine Olsson, and Michael Nielsen.
298
+
I’m also grateful to Shan Carter, Ian Johnson, Dario Amodei, Holden Karnofsky, Anna Goldie, Smitha Milli, Nick Beckstead, Arvind Satyanarayan, Yomna Nasser, Matt Hoffman, Emma Pierson, Martin Abadi, Greg Corrado, Amy McDonald, Jeff Dean, Samy Bengio, Sageev Oore, Konstantinos Bousmalis, Peter Liu, Andrew Dai, Jasper Snoek, Delip Rao, and Vincent Vanhoucke.
299
+
</p>
221
300
222
301
302
+
<p>
303
+
The wording "pool of goodwill" is inspired by Marilynne Robinson's "reservoir of goodness."
0 commit comments