Unleash the Magic of Mage Global Hooks

• What are Global Hooks

• Why Use Global Hooks

• Create a Global Hook

• Conclusion

Global Hooks in Mage are a powerful feature that allows you to execute custom code at specific points during the application’s execution cycle. These hooks can be used to perform various operations such as data validation, transformation, or integration with external systems. Global Hooks are particularly useful when you need to automate repetitive tasks or enforce certain business rules across multiple components of your application.

Global Hooks in Mage can be triggered at two different points during the execution of a pipeline:

1. Pre-completion of a Block

   : The hook will run before a specific block in the pipeline is executed. This allows you to perform operations like data validation, transformation, or enrichment before the block processes the data.

2. Post-completion of a Block

   : The hook will run after a specific block in the pipeline has completed execution. This enables you to perform actions based on the block’s output, such as integrating with external systems, logging, auditing, or triggering downstream processes.

By leveraging these two execution points, Global Hooks provide a flexible way to extend the functionality of your data pipelines. You can choose to execute custom code either before or after a block’s execution, depending on your specific requirements.

Source: Giphy

For developers, global hooks provide a powerful and flexible way to extend the functionality of their applications, promoting code reuse and maintainability. By allowing the execution of custom code at specific points, global hooks solve the problem of having to manually implement the same logic repeatedly across different components. This centralized approach saves time, effort, and promotes consistency, enabling developers to focus on core application logic while easily integrating cross-cutting concerns like logging, security, or external system integrations.

Global hooks can be useful for implementing cross-cutting concerns that span multiple components or modules, such as logging, error handling, authentication, and authorization. They can also facilitate integration with external services or APIs by providing a centralized point for making requests and processing responses. Additionally, global hooks can help enforce business rules, data validation, and transformation logic consistently across an application. In complex applications with numerous components and data flows, global hooks can promote code reuse and maintainability by encapsulating shared functionality.

Before creating a hook, developers should ensure they have toggled on their global hooks setting. From the Mage Project Overview page, select settings from the left navigation menu. Once in the settings page toggle on Global Hooks.

Lets say we own several different pipelines within a team and we wanted to write a prefix of our initials in the name of the pipeline so we could easily identify them. How would we do this?

• First open mage and create a new pipeline

• If you don’t have mage check out their docs on how you can get

  started

• Once you are in the pipeline editor page, pictured below, select the data loader block, hover over python, then click the Generic (no template) option

• Once the block loads, delete the contents and add the code below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import os
from datetime import datetime

from mage_ai.settings.repo import get_repo_path


@data_loader
def load_data(*args, **kwargs):
    with open(os.path.join(get_repo_path(), f'ping_{datetime.utcnow().timestamp()}'), 'w') as f:
        f.write('')

    payload = kwargs.get('payload', {})
    name = payload.get('name')

    payload['name'] = f'cff_{name}'

    return payload

• Replace the prefix cff with your initials or you can leave the code how it is

• Run the block by hitting shift + enter (the block should return <your initials>_None)

Mage developed an easy to use Global Hooks user interface where developers can choice select or toggle on and off most of their requirements. Let’s navigate to the Global Hooks UI by selecting global hooks from the left navigation popout in the main Pipelines page.

• Select + add new global hook

• Once you are in the Global Hooks UI create a unique name for the hook, the resource type, and the operation type.

• For each hook we need to select a resource type which are API calls in Mage. Selecting the pipeline resource means that our hook will affect a pipeline API call.

• We’re also using this hook when creating a new pipeline so let’s select the create operation type

• As mentioned above hooks run before or after an operation starts. In this case inserting our initials before a pipeline name, will take place before an operation starts.

• Next click the the pipeline to execute drop down and select the pipeline for your hook

• Click the Update Snapshot button to create a snapshot of the hook. Developers will see a green snapshot valid indicator when the snapshot is valid

• Add the block to extract data from as the block name from your project

• Select the type of output of your data that will be merged

• Finally click save changes at the bottom of the screen to save the hook

• Now, navigate to the create pipeline page and click the + New button

• Type in a new name for the pipeline

• Click Create

If you configured the Global Hook as instructed above, you should see that all your pipeline names will begin with the initial that you provided in the data loader block of the pipeline. A company could create some conformity around what they name their pipelines so that developers can easily locate the pipelines they created and own.

In conclusion, global hooks are a powerful feature that provide developers with a flexible and extensible way to enhance their applications and integrate custom functionality seamlessly. By allowing the execution of custom code at specific points in the application lifecycle, global hooks solve common problems like code duplication, cross-cutting concerns, and external system integration. From data validation and transformation to logging and monitoring, global hooks enable developers to implement a wide range of use cases consistently across their projects. Additionally, features like targeting specific components and asynchronous execution offer granular control and performance optimization. For more information on

.