Mage Battlegrounds: Craft insights from real-time customer behavior analysis
Shashank Mishra
Data Engineering
TLDR
Join us for our first ever data engineering community competition! This 24-hour virtual hackathon will begin on April 15, 2023 with chances to win prizes totaling INR 82,500! We'll be kicking off the event with an opening ceremony on
YouTube live streamingon Saturday, April 15, 2023 at 10am PST (10:30pm IST).
This event has concluded. Thank you to all the amazing participants! Submissions are currently being reviewed. Winners will be announced on Tuesday, April 18. Stay tuned!
Problem Statement
Design and implement a real-time data pipeline that ingests, processes, and analyzes streaming data from multiple sources, and provides valuable insights through data visualization.
Scenario:
A multinational e-commerce company wants to track customer behavior, website performance, and inventory management in real-time. Your task is to create a data pipeline that processes streaming data from various sources like user activity, server logs, and inventory updates, and provides real-time analytics to help the company make informed decisions.
Requirements
Ingest and process streaming data from multiple sources (e.g., user activity, server logs, and inventory updates).
Implement data transformations and enrichment as needed.
Store processed data in a scalable and query-optimized storage solution.
Develop real-time analytics and visualizations to support decision-making.
Evaluation Criteria
Fully functional, working & executable code
Implementation for maximum business requirements
Scalability and performance of the data pipeline
Readable and modular code with proper commenting
Effective use of data engineering tools and technologies
Data validation and Error handling
Data engineering best practices
Quality Documentation and Presentation
How to submit?
Submit your project using the following form link:
https://forms.gle/1TjARW1Kqgp55j7D7Required information in google form : Name, Email ID, LinkedIn Profile, Github Repo Link, Drive link for video demo
Candidates are required to push their code and documentation in Github
Note
: Make sure to document your project including architecture diagram and HLD, create README files for project installation and end to end execution
Participants have to create a 3-4 minute video demo of their project
Projects must be submitted by April 16, 2023, 11am PST (11:30pm IST). Late submissions will not be accepted.
Guidelines for video demo
Candidates are required to create a video demo for project presentation
Length of the video should not be greater than 4 minutes
Audio should be clear and working demo of project should be recorded
Mandatory - Video Face cam is also required while recording project demo, refer
share your screenexample
Data Engineering Tools and Technologies
Data ingestion: Apache Kafka (
https://kafka.apache.org/) , Apache Nifi (
https://nifi.apache.org/) , or AWS Kinesis (
https://aws.amazon.com/kinesis/)
Data processing: Apache Flink (
https://flink.apache.org/), Apache Beam (
https://beam.apache.org/), or Apache Spark Streaming (
https://spark.apache.org/streaming/)
Storage solutions: Apache Cassandra (
https://cassandra.apache.org/), Google Bigtable (
https://cloud.google.com/bigtable), or Amazon DynamoDB (
https://aws.amazon.com/dynamodb/)
Real-time analytics: Elasticsearch (
https://www.elastic.co/) or Apache Druid (
https://druid.apache.org/)
Data visualization: Grafana (
https://grafana.com/), Kibana (
https://www.elastic.co/kibana), or Apache Superset (
https://superset.apache.org/)
Tutorials and Examples
Building Real-Time Data Pipelines with Apache Kafka (
https://www.confluent.io/blog/stream-data-platform-1/)
Real-Time Data Processing Using Apache Flink (
https://www.toptal.com/apache/apache-flink-stream-processing)
Real-Time Analytics and Visualization with Elasticsearch and Kibana (
https://www.elastic.co/webinars/getting-started-with-elasticsearch-and-kibana)
Event details
You're invited to participate in the very first
Mage Battlegrounds: Craft insights from real-time customer behavior analysis
, a 24-hour virtual hackathon hosted by
Shashank Mishra! This data engineering competition will take place on
Saturday, April 15, 2023 beginning at 11am PST (11:30pm IST)
. This will be a global event open to all participants who
register.
Starts Saturday, April 15, 2023 | 11am PST (11:30pm IST) with an opening ceremony and ends Sunday, April 16, 2023 | 11am PST (11:30pm IST) with a closing ceremony.
Winners of the competition will be selected by Tuesday, April 18, 2023 and all participants will receive an email with the announcement of the results and any closing remarks. Fulfillment of prizes will be followed in the following weeks.
- Register here
. Registration will be closed at 11am PST (11:30pm IST) on Saturday, April 15, 2023.
You may register as a solo participant or a team of max two (2) members. Upon registration, you will receive a confirmation and a follow-up email on the day of the event with details on the challenge, requirements, and additional resources to help you successfully complete and submit your project.
Opening ceremony will begin on Saturday, April 15, 2023 at 10am PST (10:30pm IST). Join via
YouTube live streaming. During this time, you will be greeted by our host, Shashank Mishra and we will present the problem statement and provide further details about the competition. We encourage all participants to attend this important event, as there will be time for questions prior to the hackathon commencement.
You can check out Mage
here.
For questions prior to this event, please visit the Mage community slack:
https://www.mage.ai/chat(Make sure to join the #battlegrounds channel).
Further details will be announced during the opening ceremony and emails will also be sent to registered participants throughout the event.
Prizes
The following prizes will be awarded for this event:
Seven (7) Amazon gift cards (valued at INR 2,500 each) & Mage swag for honorable mentions (Loot).
Cash prize of INR 10,000 & Mage swag for 2nd Runner up (Grand Mage).
Cash prize of INR 20,000 & Mage swag for 1st Runner up (Sorcerer Supreme).
Cash prize of INR 35,000 & Mage swag for the Grand Prize winner (Archmage).
Start building for free
No need for a credit card to get started.
Trying out Mage to build ranking models won’t cost a cent.
No need for a credit card to get started. Trying out Mage to build ranking models won’t cost a cent.