Mage Battlegrounds: Craft insights from real-time customer behavior analysis

This event has concluded. Thank you to all the amazing participants! Submissions are currently being reviewed. Winners will be announced on Tuesday, April 18. Stay tuned!

Design and implement a real-time data pipeline that ingests, processes, and analyzes streaming data from multiple sources, and provides valuable insights through data visualization.

Scenario:

A multinational e-commerce company wants to track customer behavior, website performance, and inventory management in real-time. Your task is to create a data pipeline that processes streaming data from various sources like user activity, server logs, and inventory updates, and provides real-time analytics to help the company make informed decisions.

1. Ingest and process streaming data from multiple sources (e.g., user activity, server logs, and inventory updates).

2. Implement data transformations and enrichment as needed.

3. Store processed data in a scalable and query-optimized storage solution.

4. Develop real-time analytics and visualizations to support decision-making.

1. Fully functional, working & executable code

2. Implementation for maximum business requirements

3. Scalability and performance of the data pipeline

4. Readable and modular code with proper commenting

5. Effective use of data engineering tools and technologies

6. Data validation and Error handling

7. Data engineering best practices

8. Quality Documentation and Presentation

• Submit your project using the following form link:

  https://forms.gle/1TjARW1Kqgp55j7D7

  • Required information in google form : Name, Email ID, LinkedIn Profile, Github Repo Link, Drive link for video demo

• Candidates are required to push their code and documentation in Github

• Note

  : Make sure to document your project including architecture diagram and HLD, create README files for project installation and end to end execution

• Participants have to create a 3-4 minute video demo of their project

• Projects must be submitted by April 16, 2023, 11am PST (11:30pm IST). Late submissions will not be accepted.

• Candidates are required to create a video demo for project presentation

• Length of the video should not be greater than 4 minutes

• Audio should be clear and working demo of project should be recorded

• Mandatory - Video Face cam is also required while recording project demo, refer

  share your screen

  example

1. Data ingestion: Apache Kafka (

   https://kafka.apache.org/

   ) , Apache Nifi (

   https://nifi.apache.org/

   ) , or AWS Kinesis (

   https://aws.amazon.com/kinesis/

   )

2. Data processing: Apache Flink (

   https://flink.apache.org/

    ), Apache Beam (

   https://beam.apache.org/

    ), or Apache Spark Streaming (

   https://spark.apache.org/streaming/

    )

3. Storage solutions: Apache Cassandra (

   https://cassandra.apache.org/

    ), Google Bigtable (

   https://cloud.google.com/bigtable

    ), or Amazon DynamoDB (

   https://aws.amazon.com/dynamodb/

    )

4. Real-time analytics: Elasticsearch (

   https://www.elastic.co/

    ) or Apache Druid (

   https://druid.apache.org/

    )

5. Data visualization: Grafana (

   https://grafana.com/

    ), Kibana (

   https://www.elastic.co/kibana

    ), or Apache Superset (

   https://superset.apache.org/

   )

1. Building Real-Time Data Pipelines with Apache Kafka (

   https://www.confluent.io/blog/stream-data-platform-1/

   )

2. Real-Time Data Processing Using Apache Flink (

   https://www.toptal.com/apache/apache-flink-stream-processing

   )

3. Real-Time Analytics and Visualization with Elasticsearch and Kibana (

   https://www.elastic.co/webinars/getting-started-with-elasticsearch-and-kibana

   )

You're invited to participate in the very first

Mage Battlegrounds: Craft insights from real-time customer behavior analysis

, a 24-hour virtual hackathon hosted by

! This data engineering competition will take place on

Saturday, April 15, 2023 beginning at 11am PST (11:30pm IST)

. This will be a global event open to all participants who

.

• Starts Saturday, April 15, 2023 | 11am PST (11:30pm IST) with an opening ceremony and ends Sunday, April 16, 2023 | 11am PST (11:30pm IST) with a closing ceremony.

• Winners of the competition will be selected by Tuesday, April 18, 2023 and all participants will receive an email with the announcement of the results and any closing remarks. Fulfillment of prizes will be followed in the following weeks.

• Register here

  . Registration will be closed at 11am PST (11:30pm IST) on Saturday, April 15, 2023.

• You may register as a solo participant or a team of max two (2) members. Upon registration, you will receive a confirmation and a follow-up email on the day of the event with details on the challenge, requirements, and additional resources to help you successfully complete and submit your project.

• Opening ceremony will begin on Saturday, April 15, 2023 at 10am PST (10:30pm IST). Join via

  YouTube live streaming

  . During this time, you will be greeted by our host, Shashank Mishra and we will present the problem statement and provide further details about the competition. We encourage all participants to attend this important event, as there will be time for questions prior to the hackathon commencement.

• You can check out Mage

  here

  .

• For questions prior to this event, please visit the Mage community slack:

  https://www.mage.ai/chat

  (Make sure to join the #battlegrounds channel).

• Further details will be announced during the opening ceremony and emails will also be sent to registered participants throughout the event.

The following prizes will be awarded for this event:

• Seven (7) Amazon gift cards (valued at INR 2,500 each) & Mage swag for honorable mentions (Loot).

• Cash prize of INR 10,000 & Mage swag for 2nd Runner up (Grand Mage).

• Cash prize of INR 20,000 & Mage swag for 1st Runner up (Sorcerer Supreme).

• Cash prize of INR 35,000 & Mage swag for the Grand Prize winner (Archmage).