05 Mar Experiencing a CRISP-DM and Agile fusion @viesure
We @viesure are very passionate about agile working practices and most of our awesome teams employ SCRUM. In viesure’s AI Team we pride ourselves on working on complete products, starting with the idea, all the way to the deployable Docker container. This requires us to combine best practices stemming from both AI and software engineering. A good example is that we follow the CRISP-DM process model for our data science projects as well as an agile software development process based on SCRUM.
CRISP-DM stands for Cross Industry Standard Process for Data Mining and is a 1996 methodology created to shape Data Mining projects. Read more about CRISP-DM at https://www.datascience-pm.com/crisp-dm-2.
The CRISP-DM process model has six phases starting with (1) Business and (2) Data Understanding, followed by (3) Data Preparation and (4) Modelling, onto (5) Evaluation and eventually (6) Deployment. CRISP-DM suggests cyclic iterations through the phases and some phases are best executed in parallel.
We use these phases as starting points for our combined efforts. We employ a Kanban board to organize our tasks and operate on a user story level. Each user story is a simple description of a requirement focussing on the value for the user. Our Product Owner ensures that our user stories are written after asking the question: Who wants what why? We map each user story to the according phase of the CRISP-DM process and put the phase name in the story title. Of course, there are some stories that don’t fit CRISP-DM such as organizational and research stories. We label these stories with their respective types and this helps us to keep a complete picture of our tasks on our Kanban board. This comes in handy for our reporting efforts where we need to explain the (non-)magic of AI.
We have a very supporting SCRUM master on our side who helps us with his knowledge and experience of agile methods. Our SCRUM master is also our first contact person when we need to adapt our current working model. We took over some elements directly from SCRUM such as the two-week-long sprints as most other teams @viesure and retrospectives. At the beginning of each sprint, we sit down together, review our achieved work and plan the tasks for the next two weeks. We use retrospectives to learn from past mistakes. Some elements from SCRUM that are harder to apply in data science are estimations and expectation management with regards to sprint results. In our experience, it’s much harder to estimate effort and complexity for our data science stories than for the software engineering ones, because understanding the data better will quite often open up Pandora’s box and a lot more questions will arise. It’ll definitely never get boring @viesure! 😉