The process of creating data architecture is divided into two key stages. Let's look at each of them in more detail.
Preparation
Before starting development, a data scientist must understand three key aspects:
What type of information will be stored?
At this stage, the best ways to organize the data flow are determined, the appropriate database type and database management system (DBMS) are selected. For example, for an online store, it may be optimal to use a relational model. In turn, a manufacturer of goods may choose a document-oriented database for storing unstructured data, such as descriptions and instructions.
How will the information be archived?
At this stage, the methods of cleaning and the form of data normalization are determined. For example, it is necessary to determine which records to delete and what tools will be used to implement this (built-in functions, manually, or user scripts).
How will the data be used?
In most cases, end users (business representatives) do not have knowledge of programming languages or SQL. That is why it is necessary to develop a graphical interface or dashboard to visualize the required information.
To answer these questions, a data scientist consults with customers, analysts, and data scientists.
Download a free selection of tools for calculating KPIs and increasing marketing metrics
Alexander Kuleshov
Alexander Kuleshov
General Director of Sales Generator LLC
Read more posts on my personal blog:
Over the past 7 years, we have conducted usa student data package over 23,000 comprehensive website audits and I have learned that all of us as leaders need clear and working algorithms for our marketing and sales.
Today we will share with you 6 of the most valuable documents that we have developed for our clients.
Download for free and implement today:
Step-by-step guide to creating marketing KPIs
Template for calculating KPIs for a marketer
9 Examples of Universal Selling Commercial Proposals
Upgrade your CPs to close more deals
How to make KPI for the sales department so that profits grow by 20% or more?
Step-by-step template for calculating KPIs for OP managers
Checklist of 12 main indicators for website promotion
Find out what metrics are needed to properly optimize your website
40 Services for Working with Blog Content
We have collected the best services for working with content
How to define your target audience without mistakes?
A proven guide to defining a company's target audience
Download the collection for free
pdf 8.3 mb
doc 3.4 mb
Already downloaded
153318
Development of data architecture
When designing a lambda architecture, it is necessary to describe it at three levels:
Business layer (external)
A system is developed to support the business in its work. This level describes how the company's employees will interact with the database to make decisions. Different departments may have different interfaces and methodologies. For example, the logistics department may monitor delivery parameters and warehouse status, while marketers focus on analyzing web traffic and the effectiveness of advertising campaigns.
At this level, depending on the business objectives, the concept of the database model and analysis methods are described.
The availability of trained specialists on the company's staff is another factor that is taken into account when developing lambda architecture. For example, an analyst is needed to effectively work with the company's data.
Conceptual level
Describes the main characteristics of the database - data types, their attributes and relationships, as well as processing methods. This level represents the database administrator's view of data organization.
This level also describes:
methods of working with architecture, including instructions, diagrams, or a set of skills required to train employees;
security requirements, such as which specialists have the right to edit data.
Technical level (internal)
Parameters related to physical data storage (volume, location and required resources) are described. For example, server hardware and software required to organize optimal information storage are determined.
Case: VT-metall
Find out how we reduced the cost of