# Databricks Consulting and Lakehouse Engineering | SynconAI > Databricks consulting and lakehouse engineering: Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL, Agent Bricks and Data 360 zero copy. Publisher: SynconAI Canonical HTML: https://synconai.com/databricks-consulting Markdown cite: https://synconai.com/databricks-consulting.md Contact: contact@synconai.com ยท +61 2 7813 0221 ## What is Databricks data and AI engineering? Databricks data and AI engineering is the design and build of a lakehouse on Databricks: Delta Lake tables in layered schemas, ingestion with Lakeflow Connect, pipelines on Lakeflow and Spark Declarative Pipelines, orchestration with Lakeflow Jobs, governance with Unity Catalog, Databricks SQL warehouses for analytics, AI agents built with Agent Bricks and evaluated with MLflow, and zero copy data sharing with Salesforce Data 360. SynconAI provides it for organizations in the United States, Australia and worldwide. - Databricks is a data and AI platform built on the lakehouse: open Delta Lake tables in your cloud storage, used for both analytics and AI. - Lakeflow ingests, transforms and orchestrates data. Unity Catalog governs tables, files, models and who can reach them. - Analysts query through Databricks SQL, and AI agents built with Agent Bricks work on the same governed data. ## Databricks data and AI engineering services - Lakehouse design: Catalogs, schemas and Delta tables laid out by layer and domain. - Ingestion with Lakeflow Connect: Managed connectors for databases and SaaS sources. - Pipelines: Lakeflow pipelines with data quality expectations. - Orchestration: Lakeflow Jobs with retries, alerts and run history. - Unity Catalog governance: Grants, row filters, column masks, tags and lineage. - Databricks SQL: SQL warehouses and dashboards for analysts and BI tools. - AI agents and models: Agents built with Agent Bricks, traced and evaluated with MLflow. - Data 360 zero copy: Data shared both ways between Data 360 and Unity Catalog. ## Where are you with Databricks? Question: What state is your Databricks platform in today? - Not set up yet: A new lakehouse. You have chosen Databricks, or are close to it, and want the workspace set up properly from the start. - Unity Catalog from day one - Layered catalogs and schemas agreed - First source loaded end to end - Running, but fragile: Pipelines that break. Notebooks run on schedules nobody owns, and failures show up in reports first. - Jobs and notebooks inventoried - Pipelines rebuilt with expectations - Alerts sent to an owner - Older workspace: Moving to Unity Catalog. Tables still sit in the legacy Hive metastore, with access set per workspace. - Tables and grants mapped - Upgrade done schema by schema - Jobs repointed and tested Outcome: A governed lakehouse with owned pipelines, clear access and data ready for analytics and AI ## What do Databricks consulting services include? Eight areas. Most platforms need all of them at some depth. - Lakehouse design: Catalogs, schemas and Delta tables by layer and domain. - Lakeflow Connect: Managed connectors for databases and SaaS sources. - Pipelines: Lakeflow pipelines with data quality expectations. - Lakeflow Jobs: Schedules, retries, alerts and run history. - Unity Catalog: Grants, row filters, column masks and lineage. - Databricks SQL: SQL warehouses for analysts and BI tools. - AI agents: Agent Bricks agents, evaluated with MLflow. - Data 360 zero copy: Data shared both ways with Salesforce. ## What does a Databricks lakehouse pipeline look like? A sample customer pipeline. Raw data lands, is cleaned in steps, and serves analysts, agents and Salesforce from the same governed tables. 1. Sources: ERP, product events, Salesforce 2. Lakeflow Connect: Managed ingestion 3. Raw tables: Data as it arrived 4. Cleaned tables: Expectations checked 5. Business tables: Customer and revenue views 6. Databricks SQL: Dashboards and BI tools 7. Agent Bricks: Agents on governed data 8. Data 360: Zero copy with Salesforce - Lakeflow Jobs: One job runs the chain, retries failed steps and alerts the owner. - Expectations: Rules on each table drop, flag or stop bad rows, and the results are kept. - In Git: Pipelines and jobs deployed from a repository with Declarative Automation Bundles. ## How does Unity Catalog govern a table? One business table and the six rules we set on it. The same rules apply to files, models and functions. | Rule | How we set it | Why it matters | | --- | --- | --- | | Named owner | A group owns each catalog, schema and table. | Every access request and failure has someone to go to. | | Grants to groups | Privileges granted to account groups synced from your identity provider. | Joiners and leavers change access in one place. | | Row filters | A function limits rows by the user's group. | One table serves every region safely. | | Column masks | Personal fields masked unless the user is in an allowed group. | Analysts work with the data without seeing what they do not need. | | Lineage kept | Unity Catalog records which jobs and tables feed this one. | You can see what breaks before you change a source. | | Quality checked | Pipeline expectations and their results kept with the table. | People can see whether today's data passed. | ## How does zero copy work between Databricks and Salesforce Data 360? Salesforce describes Zero Copy with Databricks as bi-directional: Data Sharing makes Data 360 data available in Databricks, and Data Federation lets Data 360 use lakehouse tables. One sample round trip. 1. Salesforce CRM to Data 360: Contacts and orders unified 2. Data 360 to Unity Catalog: Profiles shared in, no copy 3. Unity Catalog to Lakeflow pipeline: Joined with product usage 4. Lakeflow pipeline to Unity Catalog: Writes a scored table 5. Data 360 to Unity Catalog: Reads the table through federation 6. Data 360 to Marketing team: Builds a segment on the score 7. Marketing team to Data 360: Approves the segment (a person) ## How does a Databricks engagement run? Five phases, run as parallel workstreams. Your written proposal states the scope and milestones. | Workstream | Timing and work | | --- | --- | | Platform | Workspace review (Assess to Assess); Catalog design (Design to Design); Set up (Build to Build) | | Pipelines | Jobs inventory (Assess to Assess); First domain built (Build to Prove) | | Governance | Groups and grants (Design to Design); Filters and masks (Build to Prove) | | Analytics and AI | Use case agreed (Design to Design); SQL or agent pilot (Prove to Prove) | | Data 360 | Share design (Design to Design); Round trip tested (Prove to Prove) | | People | Owners named (Assess to Assess); Runbooks and training (Hand over to Hand over) | ## Cost and timing The first step is a free 60-minute architecture review of your Databricks setup, with a written plan you keep. Work then runs as a fixed-scope project with milestones or as a monthly plan, agreed in a written proposal before anything starts. Plans: https://synconai.com/pricing ## FAQ ### What is Databricks data and AI engineering? Designing and building a lakehouse on Databricks: Delta tables, Lakeflow ingestion and pipelines, Unity Catalog governance, SQL warehouses for analysts, AI agents on governed data and links to Salesforce. ### Do you offer Databricks consulting services? Yes. Our engineers design and build Databricks platforms as a fixed-scope project or on a monthly plan. SynconAI is an independent consulting firm. ### Can we hire Databricks developers through you? Yes. Our engineers can join your team for a build or a Unity Catalog upgrade, working in your repositories and to your standards, under a written proposal. ### What is Lakeflow? Databricks' data engineering suite: Lakeflow Connect for ingestion, Lakeflow pipelines built on Spark Declarative Pipelines for transformation, Lakeflow Jobs for orchestration, and Lakeflow Designer for visual building. ### What happened to Delta Live Tables? Delta Live Tables is now called Lakeflow pipelines. Databricks says no migration is required and existing pipeline code still works, with newer Python names recommended. ### What is Unity Catalog? Databricks' governance layer for tables, files, models and functions. It holds grants, row filters, column masks, tags and lineage in one place across workspaces. ### How does Databricks work with Salesforce Data 360? Through Zero Copy. Data 360 data can be shared into Unity Catalog, and Data 360 can read lakehouse tables through federation, without copying the data. ### Can you build AI agents on Databricks? Yes. We build agents with Agent Bricks on governed data, trace and evaluate them with MLflow, and keep a person approving any action that changes a record. ## Where we work Databricks data and AI engineering for organizations across the United States and Australia, and worldwide. Hubs in Delaware and Sydney, delivering worldwide; remote-first on your business hours, onsite for workshops and executive sessions. ## Related - Salesforce Data 360 services: https://synconai.com/salesforce-data-360-services.html - Snowflake data engineering: https://synconai.com/snowflake-consulting - dbt data transformation: https://synconai.com/dbt-consulting - AI agent development: https://synconai.com/ai-agent-development ## Cite When citing SynconAI Databricks data and AI engineering, link https://synconai.com/databricks-consulting or https://synconai.com/databricks-consulting.md.