Data Recording in the Laboratory
...way forward is simply giving very well-intentioned people a system that fosters their ability to produce the very best, very most trustworthy science
Stephen Lindemann, PhD
Chief Scientific Officer, Liminal Inc.
Brad – if you ever read this, I’m sorry: data recording in the laboratory
“I'm practically perfect from head to toe
If I had a fault it would never dare to show
I'm so practically perfect in every way”
That is, at least what I desperately wanted to be.
As we have said many times before – we believe that the vast majority of the players in academic science are very well-intentioned. Certainly, I was. However, now that I am in the principal investigator (PI) seat myself, I can clearly see where my good intentions interacted with my training to produce a perverse outcome.
I was always really, really good at science. I was the kid who always knew the answer in class, who fidgeted in his seat with his hand up waiting to be called. In high school, I took every AP science class offered. And I was an inquisitive pain in the ass, always wanting to know the why of things. Once I asked Mr. Edington what the function of a particular protein structure he displayed from a laserdisc on an ancient cathode ray tube TV that was wheeled into the room. I don’t think I had ever seen a ribbon diagram before. Say what you will, I was naïve - it captivated me, the way that the structure of something as small as a protein could be determined and visualized. I asked Mr. Edington what that protein did. Rightly or wrongly, his response to me was “nobody knows.” In retrospect, I think probably somebody did know – nobody makes an X-ray crystal structure for a protein without a putative function. But I believed him. And I took the lack of human knowledge personally, as a challenge for me to solve. I’d spent my childhood planning to be an aerospace engineer – hell, at 5 I’d sent some hand-drawn space station plans to NASA. To their great credit – and something I cannot imagine happening nowadays – they sent me a big packet full of space spice for a five-year-old brain. I remember watching the last Challenger launch on the foot of my parent’s bed on a tiny TV. I can still see the images – or some mentally massaged version of them – in my mind today. I was convinced I would work at NASA and design space shuttles to prevent anything like that from happening again.
Then I saw a protein structure. And with that one last hit, I was hooked - from then on, I was a biology junkie. In college, I was the guy who showed up at office hours to not to ask about the exams but to ask other random questions that piqued my curiosity (chiefly, the aforementioned Dr. Peter Hollenbeck). I took every biology class I could fit into my schedule. In turn, I went on to be a teaching intern with Peter, sitting alone in my office hours most of the time except for exam weeks and wondering why other students weren’t as curious (or proactive). I berated myself for missing occasional points on biology exams. I performed undergraduate research for three and a half years and stayed in the lab two summers. I graduated as an Outstanding Senior in Purdue’s College of Science in 2004. I was the Student Responder at Commencement. And I did what great students do – I went to grad school.
I brought all of that built-up perfectionism into grad school. I was still my top student self in classes, where the content to master is clear and the expectations defined. In the lab, my perfectionism manifested in a very unexpected and negative way. This is because I viewed my lab notebook as a finished record, a beautiful and complete depiction of my research. Because of that, I accidentally fudged it.
I didn’t make things up or fake data. Again, well-intentioned. What I did was delay recording in my notebook – the finished and final and beautiful record – until I had the time to make it so. So that every i was dotted and t was crossed, all pictures neatly taped in and a purpose, method, result and conclusion on every experiment. To do that, I recorded the actual data on whatever was handy – Post-It notes. Random scraps of paper. Even once the Styrofoam lid of a cuvette box. From these notes, I would later assemble the perfect record when I could find the time. The problem is – sometimes that would be later that day. Sometimes it would be the next day. And sometimes it would be even later than that. Rarely, I lost some of the little scraps of paper and sometimes would stumble on them much later. I did my best to remember exactly what I did and record it perfectly. But at some points I was not 100% sure what I did and, therefore, to write. So I did the best I could with what I had – I wrote what I was pretty sure I’d done. I am certain I missed important deviations from protocol and observations from the experiments. I am certain that some of my recordings were, maybe in small ways and maybe in large ones, different from what actually happened. They were substantially true, I am sure. But I am also sure they were at least somewhat false.
I was very well-intentioned. But those intentions interacted with my fear of producing something less than perfect to, perversely, produce a less-than-perfectly-accurate research record. Brad Jones was my Ph.D. advisor, and I’ll bet that my notebook still sits in its orange binder on the top shelf above my former bench in his lab. I doubt anyone has ever looked at it since I left. But Brad – if you’re reading this, I’m sorry. Knowing what I know and being who I am now, I’d do it very differently.
There are many reasons that research records don’t match actual work in the academic lab. Sometimes trainees are like me – perfectionistic, uncomfortable with sloppy or unfinished notebooks, wanting to present the perfect notebooks like the perfect students they are. Graduate schools are full of people like this, the excellent students looking for the “next step.” Of course, there is also the other extreme – the disorganized, the sloppy, the lazy students are there too. Most fall within the middle of the spectrum; maybe the distribution is slanted towards the perfectionistic side somewhat. The vast majority are very well-intentioned. But, like me, they are producing at least somewhat false research records. And they are recording them in a huge diversity of places – sometimes their lab-issued notebooks. Otherwise, personal notebooks. Personal computers. And yes, scraps of paper and Post-Its. Here’s to hoping no others have stooped low enough to write on the cuvette box lid.
Rather than being a perfect research record, the proper research notebook is a living document. It is the place where you record everything that happens, and when it happens. It is often messy. There are cross-outs, there are corrections. I learned all of this later, in my postdoc, working at a national lab with government-issued bound and numbered notebook with a signature line on every page. My perfectionistic habits didn’t just die overnight. I still sometimes wrote things on scraps and Post-Its. But over time my view of the notebook changed. I began to see it as a document that was perfect when it was imperfect, when it was real. I began to understand that if you don’t have time to record the experiment, you don’t have time to do the experiment. That science depends on the accuracy, not the beauty of the records.
I doubt anyone has looked at my postdoc notebooks either since I left for my faculty position. But at least they are a set of records I would still implicitly trust.
The modern laboratory notebook system does generally not reinforce the value of immediacy in recording experimental data. Even in my lab, where we have adopted a national lab-like system of bound, numbered notebooks, students still record on all kinds of platforms, some paper and some electronic. But the older and busier I get, the less I trust my memory of things in general and science in specific and the more I depend on written records. Even with my prodding for students to bring their lab notebooks to meetings, they often would bring a diversity of other notes alongside, sometimes paper, sometimes electronic. I can’t be in the lab all the time to police where they record; as we said last week, I am in the office all week, writing grants and papers, working my science small business CEO life. The students will write where they will write. We must optimize the system to get the outcome we want and need.
But immediacy is the sina qua non of data recording. Even for the smartest among us, our memories are short-lived and increasingly imperfect as time marches on. To have a faithful, trustworthy record, we must have an immediate record. Without this, the data become questionable. We must also have a complete record – one that incorporates all the relevant information and the associated data, observations, methods, and all the deviations from protocol. Without this, the data become uninterpretable.
To do this, we have to align the incentives of students with that of the broader lab. We have to make recording in the proper way as easy as possible. We have to make it as easy as possible to include all the associated records. As we have tested Liminal in my lab, one part of the culture we are trying to change is how we use the bound, paper notebook. We ask students to use this not as a final document but as their scratch pad, where they record all the data as they come hot off the instruments, alongside all the metadata (the data about the data, including the methods) to make them interpretable. We expect it to be messy. We expect that record to, by itself, be incomplete. But we expect it to be immediate. What we then do is have students snap a picture of those pages and upload them alongside their electronic record on Liminal, which serves as the more perfect, final record. Hopefully, this way the data doesn’t go missing on a random Post-It. And with AI integration that knows our lab and what we know, it makes writing a complete record easier and faster. And, hopefully, shortens the cycle on data generation and data recording.
The major problem is a cultural one. It means harmonizing our diverse pre-existing frameworks for being a student with the most effective ways to do science. For that, we hope that Liminal can serve as a system that helps and incentivizes students towards behaving in ways that inspire confidence in data across individual lab environments. Where the origin and the data are established beyond doubt. That is – the provenance of the data is easily auditable.
To us, the way forward is simply giving very well-intentioned people a system that fosters their ability to produce the very best, very most trustworthy science.
Stephen Lindemann, PhD
Chief Scientific Officer, Liminal Inc.
Very insightful! I really liked the lines: (a) "This is because I viewed my lab notebook as a finished record, a beautiful and complete depiction of my research. Because of that, I accidentally fudged it." (b) "I began to understand that if you don’t have time to record the experiment, you don’t have time to do the experiment. That science depends on the accuracy, not the beauty of the records."