Apache Airflow
Apache Airflow is an open-source tool to programmatically author, schedule, and monitor workflows. It is used by Data Engineers for orchestrating workflows or pipelines.
installation Steps: How to Install Apache Airflow? | GeeksforGeeks
know more: What is Airflow®? — Airflow Documentation
Directed Acyclic Graph
(DAG) is a group of all individual tasks that we run in an ordered fashion. In other words, we can say that a DAG is a data pipeline in airflow. In a DAG:
There is no loop
Edges are directed
Key Terminologies:
Operator: The task in your DAG is called an operator. In airflow, the nodes of the DAG can be called an operator
Dependencies: The specified relationships between your operators are known as dependencies. In airflow, the directed edges of the DAG can be called dependencies.
Tasks: Tasks are units of work in Airflow. Each task can be an operator, a sensor, or a hook.
Task Instances: It is a run of a task at a point in time. These are runnable entities. Task Instances belong to a DagRun.
know more: How to Create First DAG in Airflow? | GeeksforGeeks
Airflow with Frappe
other way to integrate Airflow with frappe
preferred way is frappe-client as decoupled
The Frappe Client is a Python library that simplifies interacting with the Frappe framework's API. It allows you to perform various operations on Frappe data and execute server-side functions from external Python applications. This is very useful when integrating Frappe with other systems.
Key Features and Concepts
REST API Wrapper: The Frappe Client acts as a wrapper around Frappe's REST API, making it easier to send requests and handle responses.
Authentication: It handles authentication with the Frappe API, typically using API keys.
CRUD Operations: You can perform Create, Read, Update, and Delete operations on Frappe documents (DocTypes).
Calling Whitelisted Methods: Execute server-side functions that are explicitly exposed (whitelisted) by Frappe.
DocType Interaction: The core functionality revolves around working with DocTypes, which are the building blocks of data in Frappe.
Installation
Before you can use the Frappe Client, you need to install it:
pip install frappe-clientAuthentication
The Frappe Client requires authentication to interact with a Frappe instance. The most common method is using API keys (which are generated within the Frappe system).
from frappeclient import FrappeClient
# Replace with your Frappe instance details and API credentials
client = FrappeClient(
"your_frappe_site_url", # e.g., "http://localhost:8000" or "https://your-site.com"
"your_api_key", # The API Key from a Frappe user
"your_api_secret" # The API Secret associated with the API Key
)
Basic Operations
Here are some fundamental operations you can perform with the Frappe Client:
Creating a Document (Insert)
new_doc = client.insert(
"ToDo", # The DocType name
{
"description": "Buy groceries",
"due_date": "2025-01-20",
"priority": "High"
},
)
print(new_doc) # Prints the newly created document data
Retrieving a Document (Get)
doc = client.get("ToDo", "TODO00001") # Replace TODO00001 with a valid name
print(doc)
Updating a Document (Update)
updated_doc = client.update(
"ToDo",
"TODO00001", # The name of the document to update
{
"description": "Buy groceries and milk",
"priority": "Medium",
},
)
print(updated_doc)
Deleting a Document (Delete)
client.delete("ToDo", "TODO00001")
print("Document deleted")
Listing Documents (Get List)
todos = client.get_list(
"ToDo",
fields=["name", "description", "due_date"], # Specify the fields you want
filters={"priority": "High"}, # Optional: Filter criteria
limit=10 # Optional: Limit the number of results
)
print(todos)
Counting Documents (Get Count)
count = client.get_count(
"ToDo",
filters = {"status": "Open"}
)
print(f"Number of open ToDos: {count}")
Calling Whitelisted Server-Side Methods
Frappe allows you to expose Python functions from your server-side code as API endpoints. These functions must be decorated with @frappe.whitelist(). The Frappe Client can then call these methods.
Server-Side (Frappe):
# In a Frappe Python file (e.g., in a DocType's controller)
import frappe
@frappe.whitelist()
def get_server_time():
import datetime
return datetime.datetime.now().isoformat()
Client-Side (Python using Frappe Client):
server_time = client.get_value("DocType", "name_of_a_doc", "get_server_time")
print(server_time)
#or
response = client.call({
"method": "your_app.your_module.get_server_time", # Dotted path to the method
})
print(response)
Best Practices and Advanced Usage
Error Handling: Wrap your Frappe Client calls in
try...exceptblocks to handle potential errors gracefully. The client may raise exceptions for network issues, API errors, or invalid requests.Pagination: When retrieving large lists of documents, use the
limitandstartparameters inclient.get_list()to implement pagination.Filtering: Use the
filtersparameter inclient.get_list()to efficiently retrieve specific data from Frappe. You can use various operators (e.g., "=", "!=", ">", "<", "like") in your filters.Field Selection: Always specify the
fieldsparameter inclient.get_list()to retrieve only the data you need. This improves performance and reduces the amount of data transferred.File Uploads: The Frappe client supports file uploads. Refer to the library documentation for details.
Batch Operations: For performance, consider using batch operations if the Frappe API supports them, to reduce the number of individual requests.
Timeouts: You can set timeouts when initializing the client to prevent your application from hanging indefinitely if the Frappe server is unresponsive.
Data Validation: Validate the data you receive from the Frappe API to ensure it meets your expectations.
By following these guidelines, you can effectively use the Frappe Client to integrate your Python applications with Frappe, enabling seamless data exchange and automation.