Azure Data Factory V2
Hybrid data integration service that simpliflies ETL at scale
UI/UX Design
As data becoming more and more important the ability to transform data to useful information become more and more demanding. Azure Data Factory, a service built for all data integration needs and skill levels. Easily construct ETL and ELT processes code-free within the intuitive visual environment, or write your own code.
Company/
Microsoft
Role /
Lead Designer
Year /
2017-2019
Dusting off the sketchbook . . .
The goal is to create the tool that would enable user, including those with less coding knowledge to transform and orchestrate their data.
The Hybrid data integration service that simplifies extract, transform and load data at scale
A problem worth solving
Microsoft realized that the main user for big data are data analyst and data scientist, however the user in these two personas not alway be tech savy. The process of cleaning and transform data usually demand the user to have extensive technical knowledge and ability to code. There wasn't that many virtual data transformation/orchestration tool that exist in the market and certainly non from Microsoft. Before data factory V2 the non-technical have to rely on data engineer to clean the data for them which mean extra time needed.
Part of Azure portal
Pure coding experience
Can only work on one pipeline at a time
No guiding information
Persona
We can separate the user into 6 personas. In ADF v1, however, we only targeted one persona - data engineer. After comprehensive interviews with data engineers we realised that they work closely with data analyst and data scientist. There is a lot of unnecessary data transfer between these roles.
Potentail User
Current User
Potentail User
Data analyst and data scientist are usually less tech savvy, most of them prefer not to code. This gave us a goal to enable these 3 personas to collaborate more effectively.
Challenges
Flexible Integration
Accommodate all the unstructured data from anywhere
Compute agnostic
ADF can hook up all the different analytic platforms and drive BI reporting
Code free visual tool
Enable complex experiences to less tech savvy users by creating intuitive visual tools
Corroborate work space
Version control & protected editing for different personas
Process
Agile design process
Wireframe
I started making wireframe once I have some idea on the direction we want to preceded. There were 2 levels: High level wireframe which used to show stakeholder, engineers and PMs and confirm that they are happy with the direction we are moving toward. Detail wireframes were used by PM to present to users(partner) to get their feedback and reaction.
Detailed wireframes were used by PM to present to users(partner) to get their feedback and reaction.
Design Details
I created visual designs after making the detailed wireframe. One of the goals for this project was to make the help center consistent with Microsoft’s component library style. I used colours, fonts, spacing, buttons, cards, icons, etc. from Microsoft branding guidelines. In visual design process, I kept in mind keywords for simple and clean.
Overview
Azure navigator bar with the factory name
Introduction video and latest news about the product
Other option
Logout option
Tutorial video
Pipeline Canvas
Activities menu (collapse able)
Version control
Coding option
Canvas action bar
Setting (resizable)
Data Flow Canvas
Guiding layout system
Coding option
Canvas action bar
Setting (resizable)
Dataset setting tab
Dataset icon
We spend a lot of time making sure that the settings make sense to the user
Coding option
Tab for switching between settings
Pipeline Run Monitor
These buttons appear only in monitoring mode
Action bar
Status
User can copy their filter and share their filter with other user
Data Flow Monitor
Outcome
Azure Data Factory launched general availability in July 2018 and October 2019 for Data Flow feature.
"I’ve worked many years with SAP and with other clouds and Microsoft is clearly on the right track. Nothing matches it in terms of interface and user friendliness"
– Alexandre G.
“It's pretty good. Very satisfied. I thought this was easier than Nifi (compatitor). This was actually very minimal. It's only a 1 step thing”
-Participant 7
“ADF improved our productivity! It helps us a lot”
-Kapil