Highlights:

  • Organizations can utilize LakeFlow to construct data pipelines, which are software workflows capable of automatically transferring information between systems.
  • AI/BI operates with a collection of AI agents, each optimized for specific tasks.

Recently, Databricks Inc. has debuted LakeFlow and AI/BI. These two new offerings aimed at simplifying various aspects of analyzing business data to uncover valuable patterns.

LakeFlow streamlines the creation of data pipelines, while AI/BI serves as a business intelligence platform featuring an integrated artificial intelligence chatbot. Databricks introduced these innovations alongside an updated release of its Mosaic AI toolkit for developing AI models, leveraging assets acquired through a startup acquisition totaling USD 1.3 billion last year.

Live Data Pipelines

At its Data AI Summit in San Francisco recently, Databricks unveiled several new products, including LakeFlow, which integrates technology from an acquisition. Databricks acquired Arcion Inc. for over USD 100 million last year; the startup had developed a tool for seamless data movement between applications, now a key component of LakeFlow.

Organizations can leverage LakeFlow to construct data pipelines—automated software workflows designed to transfer information between systems. These pipelines frequently include data transformation processes aimed at enhancing information accessibility. Databricks affirms that LakeFlow offers robust capabilities for both functionalities.

The platform features connectors capable of retrieving data from databases and widely used business applications like Salesforce, Google Analytics, and SharePoint. These connectors are seamlessly integrated with Databricks’ Unity Catalog tool, enabling administrators to monitor employee interactions with the data processed through LakeFlow.

Once data is extracted from a system, LakeFlow can enhance it with external information, adjust its structure, and apply other alterations. Developers have the flexibility to write code in SQL or Python to execute these modifications. This capability is enabled by Delta Live Tables, Databricks’ tool for ETL (extract, transform, load) operations.

Businesses with sophisticated needs can customize LakeFlow data pipelines to handle information in nearly real time. Databricks explains that this capability is supported by Real Time Mode, which can be activated with minimal adjustments to the existing code.

Customers can oversee the performance of their data pipelines through a suite of capabilities known as LakeFlow Jobs. In case of technical issues, incident data is relayed to alerting platforms like PagerDuty so they can promptly notify administrators. LakeFlow Jobs also streamlines various other tasks, such as deploying newly developed data pipelines into production.

AI-driven Dashboards

Databricks’ second new product, AI/BI, was unveiled recently and comprises two primary components. AI/BI Dashboards is a module enabling the transformation of business data into visual representations using a low-code interface. The platform’s second component, Genie, is a chatbot that allows users to ask natural language queries about these visualizations.

“A truly intelligent BI solution needs to understand the unique semantics and nuances of a business to effectively answer questions for business users. We believe this requires a different approach than how BI software has been designed in the past — one that places an AI system at the center of the architecture and is designed to take advantage of the AI systems’ strengths as well as complementing their weaknesses to tackle the challenges of understanding and learning these nuances,” said Ali Ghodsi, Co-founder and Chief Executive Officer at Databricks.

Behind the scenes, AI/BI harnesses a suite of specialized AI agents, each tailored for distinct tasks. One agent focuses on visualizing user data, another aids in interpreting information, and Databricks has developed an additional AI agent specifically designed to explain the outcomes of analytics projects.

Databricks asserts that its agents continuously learn from the data they handle. For instance, when a user defines how AI/BI should interpret terms like ‘churn’ in customer retention analysis, the platform can apply this knowledge in future analytics endeavors. Additionally, Databricks emphasizes that its AI agents glean insights from other data and technical resources within a company’s analytics ecosystem.

Similar to Dataflow, AI/BI is compatible with the company’s Unity Catalog product. Administrators can utilize Unity Catalog to oversee user interactions with business data within AI/BI. Databricks highlights that Unity Catalog serves a dual role as a tool for ensuring data reliability, enabling employees to verify the accuracy of information used in analytics projects.

Alongside data visualizations, AI/BI includes an integrated chatbot named Genie, offering employees the ability to query data using natural language. Databricks emphasizes that Genie can enhance its output quality through learning from user feedback.