The choice of technologies is an essential part of the design and architecture process of building a data analytics solution. The selected tools need to respond to the business needs. Therefore, it is crucial to spend time gaining a deep understanding of these needs and also the business rules that exist in your organization. The result should be a set of technologies and tools that are well-integrated in a system to fulfil business needs, not the other way around. If the people who use the system must adapt to tool limitations, the choices made at the design stage were obviously not the best ones. This is not an easy task as there is no perfect tool. When making up your mind you need to consider the advantages and disadvantages of every tool, see the situations and workflows it is most useful for and only then select those that fit your organization’s practices, rules, and needs the most. To help you get started, here we share our ultimate list of the technologies and platforms that enable data analytics.
Cloud platforms
Every system needs a platform to run. Today, a highly popular choice is to deploy your system to the cloud. While in the past having a big and complex system meant that you need to set up your own data center with your own physical machines, today this is no longer the case. Cloud platforms enable you to deploy and run big and complex systems on data centers, infrastructure, and services, provided and maintained by the cloud service vendors. This comes with a few benefits, including the ability to scale parts of your architecture, as the demand changes, with a few clicks, rather than having to manually add more physical machines to your center.
Microsoft Azure
Microsoft Azure is Microsoft’s public cloud computing platform. It provides a range of cloud services, including compute, analytics, storage, and networking. Users can pick and choose from these services to develop and scale new applications or run existing applications in the public cloud.
The Azure platform aims to help businesses manage challenges and meet their organizational goals. It offers tools that support all industries, including e-commerce, finance, and a variety of Fortune 500 companies. It is also compatible with open source technologies which provides users with the flexibility to use their preferred tools and technologies. In addition, Azure offers 4 different forms of cloud computing: infrastructure as a service (IaaS), platform as a service (PaaS), software as a service (SaaS) and serverless.
Microsoft charges for their Azure services on a pay-as-you-go basis, meaning subscribers receive a bill each month for the specific resources they have used.
Amazon Web Services
Amazon Web Services (AWS) is made up of many different cloud computing products and services. It provides servers, storage, networking, remote computing, email, mobile development, and security. AWS can be broken into three main products: EC2, Amazon’s virtual machine service, Glacier, a low-cost cloud storage service, and S3, Amazon’s storage system. As of February 2020, one independent analyst reports AWS has over a third of the market at 32.4%.
AWS has 76 availability zones in which its servers are located. These serviced regions are divided to allow users to set geographical limits on their services (if they choose so) and provide security by diversifying the physical locations where data is held. Overall, AWS spans over 245 countries and territories.
Google Cloud Platform
Google Cloud Platform is a provider of computing resources for deploying and operating applications on the web. Its specialty is providing a place for individuals and enterprises to build and run software, and it uses the web to connect to the users of that software. Google Cloud consists of a set of physical assets, such as computers and hard disk drives, and virtual resources, including virtual machines (VMs), that are contained in Google’s data centers around the globe. Each data center location is in a region. Regions are available in Asia, Australia, Europe, North America, and South America. This distribution of resources provides several benefits, including redundancy in case of failure and reduced latency by locating resources near the clients. This distribution also introduces some rules on using and combining resources.
Data Integration Tools
Data integration is an essential part of a big data analytics solution as it allows for unification of multiple systems into a single data model that is later used for gaining knowledge and insights. In other words, it is the method for introducing data to your analytics system. Every data analytics system needs data to be analyzed and this is the area that handles its provision. There are several different approaches to achieving this goal, all quite different and thus, solving different problems. One commonly used approach is ETL (extract, transform load). ETL is a process where large volumes of the required data are extracted from various data sources and converted into a common format, defined by your organization’s data model. The data is then cleaned and loaded into specialized storage like a data warehouse or a data lake. It is then available for standard reporting and analysis purposes. Regardless of the deployment method you chose (cloud or on-premises), you need a data integration in your system.
Microsoft SQL Server Integration Services (SSIS)
Microsoft SQL Server Integration Services (SSIS) is a platform for building enterprise-level data integration and data transformations solutions. Use Integration Services to solve complex business problems by copying or downloading files, loading data warehouses, cleaning and mining data, and managing SQL Server objects and data.
Integration Services can extract and transform data from a wide variety of sources, such as XML data files, flat files, and relational data sources, and then load the data into one or more destinations.
Integration Services include a rich set of built-in tasks and transformations, graphical tools for building packages, and the Integration Services Catalog database, where you store, run, and manage packages.
You can use the graphical Integration Services tools to create solutions without writing a single line of code. You can also program the extensive Integration Services object model to create packages programmatically and code custom tasks and other package objects.
Azure Data Factory
Azure Data Factory is Azure’s cloud ETL service for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring and single-pane-of-glass monitoring and management. You can also lift and shift existing SSIS packages to Azure and run them with full compatibility in ADF. SSIS Integration Runtime offers a fully managed service, so you don’t have to worry about infrastructure management.
Xplenty
Xplenty is an ETL platform that requires no coding or deployment. It has a point-and-click interface that enables simple data integration, processing, and preparation. It also connects to a large variety of data sources and has all the capabilities you need to perform data analytics.
AWS Glue
AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. Once catalogued, your data is immediately searchable, quarriable, and available for ETL.
Cloud Data Fusion
Cloud Data Fusion is a fully managed, cloud-native, enterprise data integration service, provided by Google for quickly building and managing data pipelines.
The Cloud Data Fusion web UI allows you to build scalable data integration solutions to clean, prepare, blend, transfer, and transform data, without having to manage the infrastructure. Cloud Data Fusion is powered by the open source project CDAP.
Storage
Properly stored, even old data can offer value thanks to new analytical tools. Fortunately, data storage is more cost-effective than ever, a trend that will continue for the foreseeable future.
Microsoft SQL Server
Microsoft SQL Server is a relational database management system, or RDBMS, developed and marketed by Microsoft. Like other RDBMS software, SQL Server is built on top of SQL, a standard programming language for interacting with the relational databases. SQL server is tied to Transact-SQL, or T-SQL, the Microsoft’s implementation of SQL that adds a set of proprietary programming constructs. SQL Server can be used to build a fully operational data warehouse.
Azure SQL Data Warehouse
Azure SQL Data Warehouse (SQL DW) is a petabyte-scale MPP analytical data warehouse built on the foundation of SQL Server and run as part of the Microsoft Azure Cloud Computing Platform. Like other Cloud MPP solutions, SQL DW separates storage and compute, billing for each separately. Unlike many other analytical data warehouse solutions, SQL DW abstracts away physical machines, and represents compute power in the form of data warehouse units (DWUs). This allows users to scale compute resources seamlessly and easily at will.
Azure SQL Data Warehouse is part of the Microsoft Azure Cloud Computing Platform, which makes choosing this database a virtual no-brainer for companies which have already invested in the Microsoft technology stack.
Azure Data Lake
Azure Data Lake includes all the capabilities required to make it easy for developers, data scientists, and analysts to store data of any size, shape, and speed, and do all types of processing and analytics across platforms and languages. It removes the complexities of ingesting and storing all your data, while making it faster to get up and running with batch, streaming, and interactive analytics. Azure Data Lake works with existing IT investments for identity, management, and security for simplified data management and governance. It also integrates seamlessly with operational stores and data warehouses, so you can extend current data applications.
Amazon Redshift
Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers.
Google BigQuery
Storing and querying massive datasets can be time-consuming and expensive without the right hardware and infrastructure. BigQuery is an enterprise data warehouse that solves this problem by enabling super-fast SQL queries, using the processing power of Google’s infrastructure. You can control access to both the project and your data, based on your business needs, such as giving others the ability to view or query your data.
Analytics Tools
Once your data has been collected and loaded into your system, you are ready to start gaining insights from it. As mentioned above, data analytics is a wide and abstract term, relating to many different techniques, so this technology area can contain a lot of different tools: data can be further analyzed using machine learning, it can be visualized using different BI visualization tools, aggregated in a different way, so it can provide different insights, based on the angle you look from. Therefore, there are a lot of tools out there, offering different capabilities and making the right choice for your system is not a trivial task.
Azure Analysis Services
Azure Analysis Services is a fully managed platform as a service (PaaS) that provides enterprise-grade data models in the cloud. Use advanced mashup and modeling features to combine data from multiple data sources, define metrics, and secure your data in a single, trusted tabular semantic data model. The data model provides an easier and faster way for users to perform ad hoc data analysis using different tools.
Azure HDInsight
Azure HDInsight is a managed, full-spectrum, open-source analytics service in the cloud, designed for enterprises. It is a cloud distribution of Hadoop components. The service makes it easy, fast, and cost-effective to process massive amounts of data. You can use open-source frameworks, such as Hadoop, Apache Spark, Apache Hive, LLAP, Apache Kafka, Apache Storm, R, and more. With these frameworks you can enable a broad range of scenarios, such as extract, transform, and load (ETL), data warehousing, machine learning, and IoT.
Apache Spark
Apache Spark is a data processing framework that can quickly perform processing tasks on very large data sets and can also distribute data processing tasks across multiple computers, either on its own or in tandem with other distributed computing tools. These two qualities are key to the worlds of Big Data and Machine Learning, which require the marshalling of massive computing power to crunch through large data stores. Spark also takes some of the programming burdens of these tasks off the shoulders of developers with an easy-to-use API that abstracts away much of the grunt work of the distributed computing and big data processing.
ElasticSearch
ElasticSearch is a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic). It is famous for its simple REST APIs, distributed nature, speed, and scalability.
Power BI
Power BI is a collection of software services, apps, and connectors that work together to turn your unrelated sources of data into coherent, visually immersive, and interactive insights. Imagine that our data is an Excel spreadsheet, or a collection of cloud-based and on-premises hybrid data warehouses. Power BI lets you easily connect to your data sources, visualize, and discover the essence of it, and share it with anyone you want.
QlikView
QlikView is Qlik’s classic analytics solution for rapidly developing highly interactive guided analytics applications and dashboards, delivering insight to solve business challenges. The modern analytics era truly began with the launch of QlikView and the game-changing Associative Engine it is built on. Revolutionizing the way organizations use data with intuitive visual discovery and boasting a customer base of 36,000, QlikView put Business Intelligence (BI) into the hands of more people than ever before.
Tableau
Tableau is a powerful and fast-growing data visualization tool used in the Business Intelligence Industry. It helps simplify raw data into an easily understandable format.
Data analysis is performed quite fast with Tableau and the visualizations created are in the form of dashboards and worksheets. The data that is created using Tableau can be understood by professionals at any level in an organization. It even allows a non-technical user to create a customized dashboard.
R
R is a language and environment for statistical computing and graphics. R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible.
Azure Machine Learning Studio
Microsoft Azure Machine Learning Studio (classic) is a collaborative, drag-and-drop tool you can use to build, test, and deploy predictive analytics solutions on your data. Azure Machine Learning Studio (classic) publishes models as web services that can easily be consumed by custom apps or BI tools, such as Excel.
Machine Learning Studio (classic) is where data science, predictive analytics, cloud resources, and your data meet.
Looker
Looker is a business intelligence software and big data analytics platform that helps you explore, analyze and share real-time business analytics easily.
Orchestration Platforms
Orchestration is the automated configuration, coordination, and management of computer systems and software. While many cloud providers offer orchestration, the delivered service might not be able to address your business needs.
Kubernetes
Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes’ services, support, and tools are widely available.
Apache Airflow
Apache Airflow is a platform that allows you to automatically create, schedule and monitor workflows.
With it you can develop workflows as Directed Acyclic Graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rick set of command line utilities ease the performing of complex surgeries on DAGs. The advanced user-interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
Becoming a data-driven organization is now within reach of every company. Cloud solutions have made access to data analytics platforms much easier. Therefore, when implemented correctly, a data analytics solution provides valuable business intelligence on your processes and opens new opportunities.
With years-long experience in developing custom data analytics solutions, Accedia helps companies consolidate and maintain consistent data across the board, implement reliable reporting and predictive analysis, eliminate time-consuming and error-prone data manipulation activities, and more.