Re-defining the role of data analysts

Not a long time ago I attended a session about agile development methods for analytics. The session was fascinating and talked about how methods such as Kanban and Scrum can be applied in the domain of analytics the same way they are applied in software and web development. I started reflecting over the situations I experienced as a data analyst in the past few years and I realized that most analysts have to wear two hats due to the changing nature of the work environment, one being a developer and the other being an analyst.

On the one hand, internal and external clients constantly demand analysts to quickly provide answers to ad-hoc questions such as “How many units did we sell in Japan last year?” or answers to more complex questions comparing two groups of customers (e.g. online vs. offline shoppers), but on the other hand, they also want access to real-time information so that they wouldn’t have to bother the analysts a lot. Sometimes, companies even take the initiative to proactively automate reports and dashboards in response to a heavy workload and recurring requests of the same nature. However, one problem that I see with this situation is the confusion analysts face regarding time management as well as client expectations management. How do you deal with an urgent ad-hoc request to provide analyses, quick answers, and all sorts of quick fixes, and get involved in development projects that require long-run strategic thinking at the same time? Most analysts do not come from a development background and lack the understanding of the project or product development lifecycle, and long-run projects end up being a lower priority or even completely scrapped as a result of companies not accounting for the nature of the analyst work.

The good news is that companies can start to gradually redefine the analyst role and make it more of an “analytics professional” to better reflect the reality as it is nowadays. In addition to changing the title, here are a few possible courses of action that I assume companies will explore in the near future:

1. At the company or the department level – Companies can consider gradually involving professionals outside the mainstream analytics niche in the department. A scrum master can help to guide an agile development of analytics products, UX designers can help with dashboard design, and project managers can help to strategize projects based on the larger potential to serve stakeholder groups in the organization, and to implement new project management tools. As a first step, companies can train analysts, so they perform a dual-role, and let them job-shadow people who have similar roles in other technological departments.
2. At the team level – Companies can re-define the structure of analytics teams to resemble software and web development teams by letting certain people specialize in back-end tasks (e.g. query optimization, data frameworks such as Python’s PySpark and Pandas, and even data integration), while the other half of the people can specialize in front-end development (e.g. data visualization, dashboarding, and presentations). Both types of analysts can keep on responding to the less complicated ad-hoc requests while specializing in their niche for the ongoing development projects.
3. At the employee level – A bottom-up approach can be taken by analytics departments that can simply start with instructions to employees on how to allocate time for each task (e.g. 50% of the day is spent on development, and the other 50% is spent on ad-hoc requests). However, in moving towards priority changes, stakeholders must be informed that the team efforts are going to be dedicated to different tasks and not just to ad-hoc type of service.

One thing for sure, the role of data analysts will have to be re-defined in the near future due to the new skill-set needed to respond to the challenges of the new work environment.