Is Collaboration killing Creativity?
If collaboration is good, is more collaboration better?
Project management methodologies that have been successful in production-centric environments, e.g., agile, dev-ops, lean are increasingly being deployed in big data projects. However, big data projects are a combination of production and creative work.
Software engineering and development is arguably production-centric and well-suited to optimisation workflows. On the other hand, Science –and research in general- explores how to reach a long-term goal. Outcomes of the scientific process are highly non-linear; significant results are obtained in a similar fashion to an artist’s creative breakthrough.
To maximise the potential of data science teams, one should provide an environment that is suitable for creativity. Fortunately, that’s a well-researched area with evidence-based answers; unfortunately, these findings are often ignored.
Adequate collaboration is the most critical enabler of creativity, but not all collaboration principles are equal: how many works of art, such as paintings or books, are the products of teamwork? In short, not that many [1].
Creative “outbursts”, in research or artistic pursuits follow the common pattern of mentally reaching out to novel – and apparently unrelated – ideas to solve a problem or fulfill a vision, until they coalesce into what, to an outsider, appears as an epiphany.
Artists’ collaborative life is well documented: from circles of philosophers to Andy Warhol’s Factory and the close connections painters and writers in European capitals a few centuries ago. The most creative people experience a mixture of solitary work and external influences: the collaborative aspect having less to do with creating the work, but more with the inspiration it provides.
A number of studies on aspects as diverse as, e.g., on the ideation process [2], the quality and success Broadway shows [3] or communication vs. productivity in the workplace [4] all demonstrate the same two aspects: too much close collaboration is harmful – as it naturally leads to cliques, groupthink, and echo chambers – , while too little contact with “the outside world” also hampers creativity.
In the realm of research – academic or otherwise – that form of collaboration had been ongoing for a long time: a personal space to create (the fast disappearing office), and a collective space to make face-to-face contact [5] and exchange ideas (the fast disappearing workplace cafeteria, external seminars or conferences).
Instead of following these findings, which have been long-held best practices, recent trends have almost obliterated them: open offices, small kitchens replicated around various floors, restriction of travel budgets and constant collaborative meetings with a core team are stifling innovation.
