By Brendan Siebecker, Director of Alliances – Atlan
By Gaurav Malhotra, Options Architect – AWS
Extra individuals are utilizing information in the present day than ever earlier than, nevertheless it’s getting tougher and tougher for everybody to collaborate on the identical information. This contains information engineers and analysts, product managers, entrepreneurs, researchers, and extra.
Atlan is an AWS Companion with the Amazon Redshift Prepared designation. It has pioneered a collaborative workspace serving to trendy information groups work collectively higher. Atlan is a collaboration and orchestration layer—the glue that brings collectively your group, the instruments you like, and the info you want.
With deep integrations throughout the trendy information stack, Atlan helps groups create a single supply of reality for all of their information property. Atlan is extending its suite of integrations by in depth collaboration with Amazon Internet Providers (AWS), and now you can discover Atlan on AWS Market.
On this submit, we’ll share how corporations use Atlan and AWS to democratize their information, collaborate extra successfully, and unify all of their data and context in a single place. We’ll additionally present methods to combine Atlan and Amazon Redshift with a step-by-step walkthrough.
How Firms Use Atlan and AWS
Clients within the AWS ecosystem can profit from Atlan’s seamless integration with a set of AWS providers—together with analytics tooling comparable to Amazon Redshift, Amazon Athena, and AWS Glue—in addition to standard instruments within the trendy information stack comparable to Tableau, Apache Airflow, and dbt.
For instance, Postman (an API platform utilized by greater than 500,000 corporations worldwide) makes use of AWS and Atlan to open up their information, construct belief, and grow to be extra data-driven. That is necessary as a result of Postman’s leaders staunchly imagine that everybody within the firm ought to have the ability to entry information and achieve insights from it. Nonetheless, earlier than Atlan, their information was usually a thriller and context lived within the heads of early group members.
Prudhvi Vasa, Analytics Chief at Postman, defined the worth of democratizing and documenting information with AWS and Atlan: “We’ve been in a position to catalog and doc all of our information, which acts as a single supply of reality for our information. The outcome? Everybody is ready to discover the suitable information for his or her use case, and the info is constant throughout the board for all accessing it,” says Vasa.
“Having a dependable information basis, the place folks can discover and perceive all our information opens the opportunity of having everybody take part in analyzing information. This permits our whole firm to grow to be extra data-aware and data-driven, which is the aim for any main firm in the present day,” Vasa provides.
Atlan, AWS, and the Trendy Knowledge Stack
Atlan acts as a virtualized layer throughout a wide range of instruments within the trendy information stack. Its push- and pull-based metadata crawlers deliver metadata from completely different instruments within the information platform to construct a unified collaboration platform.
First, Atlan creates a robust search and discovery layer. It acts as a Google-like search engine for all your information, the place you may shortly uncover and entry any information asset together with all of its related context and documentation. This search helps clever key phrase recognition, highly effective search filters, sorting by relevance or reputation, and even a Cmd+Okay shortcut.
This search doesn’t simply floor information tables—it surfaces all the things about a company’s information. In in the present day’s day and age, information property aren’t simply tables. That’s why Atlan lets folks search throughout each sort of knowledge asset—enterprise intelligence (BI) dashboards, pipelines, code, fashions, queries, metrics, directed acrylic graphs (DAGs), and extra.
Second, Atlan unifies context from all of the completely different instruments in your information stack in a single place. The place does this information come from? Who makes use of it? Can I belief it? The “asset profile” in Atlan solutions questions like these with data like an information asset’s description, certification (verified, WIP, or deprecated), column previews, pattern information, and Readme. This makes it simpler to grasp every information asset (like lineage, documentation, and possession) in a single view.
Embedded Collaboration Integrations
Atlan is constructed on the premise of embedded collaboration, borrowing rules from GitHub, Figma, Superhuman, and different trendy future-of-work instruments.
Embedded collaboration is about work taking place the place you might be, with the least quantity of friction. What when you may request entry to an information asset while you get a hyperlink, and the proprietor may get the request on Slack and approve or reject it proper there?
What if, while you’re inspecting an information asset and have to report a difficulty, you would instantly set off a assist request that’s completely built-in along with your engineering group’s Jira workflow?
Embedded collaboration unifies these micro-workflows that waste time, trigger frustration, and result in instrument fatigue, turning time-consuming duties throughout a number of instruments into a number of clicks in whichever instrument you’re already utilizing.
Getting Began with Atlan and AWS
Atlan is constructed on high of AWS’s highly effective providers. By straight integrating with AWS providers like Amazon Redshift, Atlan helps information groups accomplish extra by making collaboration a seamless a part of their course of.
Within the following step-by-step information, we’ll present you methods to shortly combine Atlan with Redshift to open up a brand new world of collaboration, readability, and belief for contemporary information groups.
Comply with the steps under to determine a connection and combine Atlan with a Redshift database.
Step 1: Choose the Supply
- Log into your Atlan workspace.
- Click on on the Workflow button within the left sidebar.
- You’ll see the Market web page with the listing of sources obtainable in your workspace. Click on on New Workflow on the high proper.
- Choose Redshift from the listing of choices within the integrations tab, and click on Setup Workflow.
Step 2: Present Credentials
- To arrange a brand new connection, fill in your Redshift credentials on the Credential web page. Under is an instance:
- Hostname: examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com
- Port: 5439
- Username: myusername
- Password: xxxxxx
- Default Database: dev
- Choose the right authentication technique (primary authentication, or an IAM person or IAM position).
- Upon getting crammed within the particulars, click on on Check Authentication after which Subsequent.
Step 3: Set Up Your Configuration
- On the Connection web page, title your connection and choose the customers or teams who ought to have the ability to entry it.
- On the Metadata web page, specify any metadata you need to embody or exclude from crawling.
- Click on Run to run the crawler as soon as, or click on on Schedule & Run to schedule it for a each day, weekly, or month-to-month run.
- When you click on Run or set a schedule, the workflow will begin operating.
Atlan will crawl your Amazon Redshift occasion and ingest the entire metadata into Atlan. This course of varies relying on the scale of the warehouse you’re seeking to crawl, nevertheless it usually takes lower than half-hour.
Step 4: Uncover Your Property
Now that you just’ve efficiently related Atlan to Amazon Redshift and the Atlan crawler has ingested the metadata, you can begin discovering your property inside Atlan.
Within the instance under, an organization has 362 Redshift property inside Atlan. These are seen on the Discovery web page, filtered by the Amazon Redshift integration.
The information group can click on by to any asset to see related context, such because the column names, glossary phrases, classifications, standing, associated queries, and Readme.
They will additionally see the info asset’s lineage, which is auto-generated on the column degree for each Redshift asset. This helps information groups see the place every asset comes from and which dashboards use it.
Establishing Atlan is simple for analysts and enterprise customers alike. From connecting your tables and dashboards inside Amazon Redshift to enriching imported information property, the method doesn’t require main information engineering sources or time. Crawling occurs seamlessly and stays in sync along with your AWS cases.
For extra data, take a look at these hyperlinks:
This text was initially printed on the AWS Companion Community (APN) weblog.