Skip to content

API reference: Airflow integration#

You can use the Neptune integration with Apache Airflow to log metadata generated during DAG runs.

NeptuneLogger#

Creates a Neptune logger instance for tracking metadata during a DAG run.

Parameters

Name         Type Default    Description
api_token str, optional None User's Neptune API token. If None, the value of the NEPTUNE_API_TOKEN environment variable is used.

⚠ To keep your token secure, avoid placing it in the source code. Instead, save it as an environment variable.
project str, optional None Name of a project in the form workspace-name/project-name. If None, the value of the NEPTUNE_PROJECT environment variable is used.
**neptune_run_kwargs str, optional - Additional keyword arguments to be passed directly to the init_run() function, such as description and tags.

Note: The custom_run_id parameter is reserved in this integration. It's automatically generated based on the DAG ID.

Examples

If you have your Neptune credentials saved as environment variables, the following creates a Neptune logger with default settings:

from neptune_airflow import NeptuneLogger

with DAG(
    ...
) as dag:

    def task(**context):
        neptune_logger = NeptuneLogger()

get_run_from_context()#

Gets the run from the current task so it can be used for logging metadata within the task.

Parameters

Name Type Default Description
context dict - Apache Airflow Context received by the task.
log_context bool False Whether to log the contents of the Context keyword arguments.

Returns

Run object that can be used for logging.

Examples

with DAG(
    ...
) as dag:
    def task_1(**context):
        run = get_run_from_context(context)
        run["some_metric"] = 0.99

get_task_handler_from_context()#

Gets a namespace handler named by the ID of the current task.

You can use the handler for logging metadata within the task, using the same logging methods as for runs. The metadata will be organized under the run["task_id"] namespace inside the run.

Parameters

Name Type Default Description
context dict - Apache Airflow Context received by the task.
log_context bool False Whether to log the contents of the Context keyword arguments.

Returns

Handler object that can be used for logging.

Examples

with DAG(
    ...
) as dag:
    def task_1(**context):
        task_handler = get_task_handler_from_context(context)
        task_handler["some_metric"] = 0.99

See also

neptune-airflow repo on GitHub