mirror of
https://github.com/SkyfallWasTaken/arcade-monitor.git
synced 2024-11-22 00:43:39 +00:00
Compare commits
No commits in common. "c84b70e609843f0020a8cbf8e9162acf5300163f" and "55cc50f7211d01316f66ea44ece87a3b6a1bf0a1" have entirely different histories.
c84b70e609
...
55cc50f721
2 changed files with 197 additions and 179 deletions
196
README.md
196
README.md
|
|
@ -1,185 +1,23 @@
|
||||||
# Arcade Monitor: HC's beloved shop ~stalker~ monitor
|
# \#arcade-monitor
|
||||||
|
|
||||||
<img src="https://github.com/user-attachments/assets/3fe6199a-7d82-4620-ad40-9ab2c10044da" align="left" width="246" style="padding-right: 3rem"/>
|
**\#arcade-monitor** is a monitor for the [Hack Club Arcade Shop.](https://hackclub.com/arcade/shop/) It continuously checks for new items, stock changes, and item updates, keeping you in the loop on the stuff you want.
|
||||||
|
|
||||||
Arcade Monitor was a **monitor for the Hack Club Arcade Shop.** It continuously checked for new items, stock changes, and item updates, keeping you in the loop on the stuff you want. At its peak, it chattered away in the [#arcade-monitor channel](https://hackclub.slack.com/archives/C079RG9HJ81) to over 400 members across Slack and ntfy.
|
## Environment variables
|
||||||
|
### Secrets
|
||||||
|
- `SLACK_WEBHOOK_URL` - Slack webhook to call on updates
|
||||||
|
- `NTFY_AUTH_TOKEN` - Auth token for ntfy
|
||||||
|
|
||||||
This has been one of my most successful projects: I managed to get 400 active users of the channel, including many from Hack Club HQ, got third-party contributions to my project for the first time, and managed to nearly 60 stars, by far my most starred project on GitHub _(will you be the next stargazer? 😉)_
|
### Variables
|
||||||
|
- `ARCADE_SHOP_URL` - Shop URL to fetch
|
||||||
|
- `NTFY_URL` - URL for ntfy
|
||||||
|
- `SLACK_GROUP_ID` - ID of the Slack group to ping
|
||||||
|
|
||||||
If I were to make this project again, I'd probably port it to more platforms, such as Discord webhooks, in order to make it possible for people to use the software they want to use, rather than being forced to use Slack or ntfy :)
|
## Key-value keys
|
||||||
|
- `real_prices` - stores the real-world prices of items. equivalent to a `HashMap<String, i32>`, where `String` is the `id` parameter. prices are in USD.
|
||||||
|
- `items` - stores old items (you don't need to worry about this one)
|
||||||
|
|
||||||
Arcade Monitor is also fully open source, and licensed under the [MIT License!](https://github.com/SkyfallWasTaken/arcade-monitor/blob/main/LICENSE.md)
|
## Tech Stack
|
||||||
|
- **Cloudflare Workers** for running the monitor on the edge.
|
||||||
|
- **Rust** for the monitor's code. I love its type safety, as well as libraries such as `serde`.
|
||||||
|
|
||||||
The rest of the readme has been archived for posterity. Thank you to everyone who checked out Arcade Monitor! 🫡
|
_Fun fact: the monitor was originally written in TypeScript, but it turned out to be so buggy I just rewrote it in Rust. The new version took less time to write, has tests (unlike the TypeScript version), and has better formatted messages._
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
### Usage on Slack
|
|
||||||
_(note: you'll need to be a member of the [Hack Club Slack](https://hackclub.com/slack/) for this!)_
|
|
||||||
|
|
||||||
Using the bot with Slack is really easy! Just join the [#arcade-monitor channel](https://hackclub.slack.com/archives/C079RG9HJ81), and you'll be added to the Arcade Monitor user group. You'll now be pinged for every update!
|
|
||||||
|
|
||||||
### Usage with ntfy
|
|
||||||
|
|
||||||
You'll need [the ntfy app](https://ntfy.sh) for this!
|
|
||||||
|
|
||||||
Once you have the app, set the server URL to `https://ntfy.hackclub.app`, and the topic to `arcade-monitor`.
|
|
||||||
|
|
||||||
That's it! You'll now receive a notification for every shop update.
|
|
||||||
|
|
||||||
## A history lesson (if you're into that sort of thing :))
|
|
||||||
<details>
|
|
||||||
<summary>Click for more details...</summary>
|
|
||||||
|
|
||||||
### The first version
|
|
||||||
|
|
||||||
The first version of Arcade Monitor was a Cloudflare Worker written in TypeScript. In hindsight, a lot of the code was needlessly complicated, and the bot was _severely_ buggy. Take a look at this:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Yeah, that wasn't the greatest first impression... 😅
|
|
||||||
|
|
||||||
It also created random "updates" that were completely unclear - what had actually happened was that the _description_ had updated, but it showed a price update.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
What was worse was that Hack Club had created a new channel called [#arcade-bulletin](https://google.com) that seemingly made the bot obsolete.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
I did quickly realise, however, that the bot could track things like stock updates that were infeasible for Hack Club staff to do every time someone bought a YubiKey.
|
|
||||||
|
|
||||||
> (Also, fun fact: the YubiKeys are technically limited, but there's over 1,000 of them. The more you know 🤷♂️)
|
|
||||||
|
|
||||||
In the future, I would definitely use libraries such as Zod to validate my scraper - it would have saved _so_ much time.
|
|
||||||
|
|
||||||
### How the scraper works
|
|
||||||
|
|
||||||
It's actually surprisingly simple! Under the hood, Hack Club's website uses Next.js, and Next.js includes a `script` tag with the ID of `__NEXT_DATA__` with a bunch of useful information. That useful information just so happens to contain the Arcade Shop's items in a nice and easy-to-use format.
|
|
||||||
|
|
||||||
<details>
|
|
||||||
<summary>View the scraper's code (~50 lines of Rust)</summary>
|
|
||||||
|
|
||||||
```rust
|
|
||||||
use reqwest::Client;
|
|
||||||
use scraper::{Html, Selector};
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
use worker::*;
|
|
||||||
|
|
||||||
#[derive(Serialize, Deserialize, PartialEq, Eq, Default, Clone)]
|
|
||||||
pub struct ShopItem {
|
|
||||||
#[serde(rename = "Full Name")]
|
|
||||||
pub full_name: String,
|
|
||||||
|
|
||||||
#[serde(rename = "Description")]
|
|
||||||
pub description: Option<String>,
|
|
||||||
|
|
||||||
#[serde(rename = "Fulfillment Description")]
|
|
||||||
pub fulfillment_description: Option<String>,
|
|
||||||
|
|
||||||
#[serde(rename = "Cost Hours")]
|
|
||||||
pub price: i32,
|
|
||||||
|
|
||||||
#[serde(rename = "Stock")]
|
|
||||||
pub stock: Option<i32>,
|
|
||||||
|
|
||||||
pub id: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
pub type ShopItems = Vec<ShopItem>;
|
|
||||||
|
|
||||||
const USER_AGENT: &str = "Arcade-Monitor/1.0 (+@SkyfallWasTaken)";
|
|
||||||
|
|
||||||
pub async fn try_fetch(shop_url: Url) -> Result<ShopItems> {
|
|
||||||
let client = Client::new();
|
|
||||||
let response = client
|
|
||||||
.get(shop_url)
|
|
||||||
.header("User-Agent", USER_AGENT)
|
|
||||||
.send()
|
|
||||||
.await
|
|
||||||
.unwrap();
|
|
||||||
let doc_html = response.text().await.unwrap();
|
|
||||||
|
|
||||||
let doc = Html::parse_document(&doc_html);
|
|
||||||
let selector = Selector::parse("#__NEXT_DATA__").unwrap();
|
|
||||||
let json: serde_json::Value = serde_json::from_str(
|
|
||||||
&doc.select(&selector)
|
|
||||||
.next()
|
|
||||||
.ok_or("no #__NEXT_DATA__ element in document")?
|
|
||||||
.inner_html(),
|
|
||||||
)?;
|
|
||||||
|
|
||||||
let available_items = serde_json::from_value(
|
|
||||||
json.pointer("/props/pageProps/availableItems")
|
|
||||||
.ok_or("availableItems not found - is the ARCADE_SHOP_URL correct?")?
|
|
||||||
.clone(),
|
|
||||||
)?;
|
|
||||||
|
|
||||||
Ok(available_items)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
</details>
|
|
||||||
|
|
||||||
<details>
|
|
||||||
<summary>View the original scraper's code (30 lines of TypeScript)</summary>
|
|
||||||
<p>Note that the type definitions are incorrect.</p>
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { parse as parseHtml } from "node-html-parser";
|
|
||||||
|
|
||||||
export type ShopItem = {
|
|
||||||
name: string;
|
|
||||||
description: string;
|
|
||||||
fulfillmentDescription: string;
|
|
||||||
id: string;
|
|
||||||
imageUrl: string;
|
|
||||||
maxOrderQuantity: number;
|
|
||||||
price: number;
|
|
||||||
};
|
|
||||||
|
|
||||||
export default function parseArcadeShopHtml(html: string): ShopItem[] {
|
|
||||||
const document = parseHtml(html);
|
|
||||||
const inner = document.getElementById("\_\_NEXT_DATA\_\_")?.text!;
|
|
||||||
|
|
||||||
const items = JSON.parse(inner).props.pageProps.availableItems.map((item: any) => {
|
|
||||||
return {
|
|
||||||
name: item\["Name"],
|
|
||||||
description: item\["Description"],
|
|
||||||
fulfillmentDescription: item\["Fulfillment Description"],
|
|
||||||
id: item.id,
|
|
||||||
imageUrl: item\["Image URL"],
|
|
||||||
maxOrderQuantity: item\["Max Order Quantity"],
|
|
||||||
price: item\["Cost Hours"],
|
|
||||||
};
|
|
||||||
});
|
|
||||||
|
|
||||||
return items;
|
|
||||||
```
|
|
||||||
</details>
|
|
||||||
|
|
||||||
## v2: the Rust version
|
|
||||||
|
|
||||||
Since my code was borked anyway, I decided that I might as well just rewrite the whole thing. I decided to rewrite it in Rust! Technically, I could've made the rewrite in TypeScript and it would've been fine, but I decided to rewrite it in Rust instead for a couple reasons:
|
|
||||||
|
|
||||||
- Serde is great and reduces boilerplate
|
|
||||||
- Rust's error handling is _superb._ I would absolutely love `Result<T, E>` to come to Go or TypeScript (and actually be used by the wider community)
|
|
||||||
- I just like Rust :D
|
|
||||||
|
|
||||||
The new version took a _lot_ less time to implement (around 3 hours) - in fact, it took less time to write than the initial version written in TypeScript! Most of my code after that was for things like enhancements and new features.
|
|
||||||
|
|
||||||
Three hours later... it worked!
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Since then, most of the changes have been things like the ticket/price ratio or prettier Slack messages that use Block Kit.
|
|
||||||
|
|
||||||
Overall, I'd call the Rust rewrite a huge success!
|
|
||||||
|
|
||||||
</details>
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
_© 2024 Mahad Kalam_
|
|
||||||
|
|
||||||
_Licensed under the [MIT License](https://github.com/SkyfallWasTaken/arcade-monitor/blob/main/LICENSE.md)_
|
|
||||||
|
|
|
||||||
180
SHOWCASE.md
Normal file
180
SHOWCASE.md
Normal file
|
|
@ -0,0 +1,180 @@
|
||||||
|
# Arcade Monitor: HC's beloved shop ~stalker~ monitor
|
||||||
|
|
||||||
|
<img src="https://github.com/user-attachments/assets/3fe6199a-7d82-4620-ad40-9ab2c10044da" align="left" width="238" style="padding-right: 3rem"/>
|
||||||
|
|
||||||
|
Arcade Monitor is a **monitor for the Hack Club Arcade Shop.** It continuously checks for new items, stock changes, and item updates, keeping you in the loop on the stuff you want. It chatters away in the [#arcade-monitor channel](https://hackclub.slack.com/archives/C079RG9HJ81) to over 350 members across Slack and ntfy.
|
||||||
|
|
||||||
|
This has been one of my most successful projects: I managed to get 350 active users of the channel, including many from Hack Club HQ, got third-party contributions to my project for the first time, and managed to get 55 stars, by far my most starred project on GitHub _(will you be the next stargazer? 😉)_
|
||||||
|
|
||||||
|
If I were to make this project again, I'd probably port it to more platforms, such as Discord webhooks, in order to make it possible for people to use the software they want to use, rather than being forced to use Slack or ntfy :)
|
||||||
|
|
||||||
|
Arcade Monitor is also fully open source, and licensed under the [MIT License!](https://github.com/SkyfallWasTaken/arcade-monitor/blob/main/LICENSE.md)
|
||||||
|
|
||||||
|
## Usage on Slack
|
||||||
|
_(note: you'll need to be a member of the [Hack Club Slack](https://hackclub.com/slack/) for this!)_
|
||||||
|
|
||||||
|
Using the bot with Slack is really easy! Just join the [#arcade-monitor channel](https://hackclub.slack.com/archives/C079RG9HJ81), and you'll be added to the Arcade Monitor user group. You'll now be pinged for every update!
|
||||||
|
|
||||||
|
## Usage with ntfy
|
||||||
|
|
||||||
|
You'll need [the ntfy app](https://ntfy.sh) for this!
|
||||||
|
|
||||||
|
Once you have the app, set the server URL to `https://ntfy.hackclub.app`, and the topic to `arcade-monitor`.
|
||||||
|
|
||||||
|
That's it! You'll now receive a notification for every shop update.
|
||||||
|
|
||||||
|
## A history lesson (if you're into that sort of thing :))
|
||||||
|
<details>
|
||||||
|
<summary>Click for more details...</summary>
|
||||||
|
|
||||||
|
### The first version
|
||||||
|
|
||||||
|
The first version of Arcade Monitor was a Cloudflare Worker written in TypeScript. In hindsight, a lot of the code was needlessly complicated, and the bot was _severely_ buggy. Take a look at this:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Yeah, that wasn't the greatest first impression... 😅
|
||||||
|
|
||||||
|
It also created random "updates" that were completely unclear - what had actually happened was that the _description_ had updated, but it showed a price update.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
What was worse was that Hack Club had created a new channel called [#arcade-bulletin](https://google.com) that seemingly made the bot obsolete.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
I did quickly realise, however, that the bot could track things like stock updates that were infeasible for Hack Club staff to do every time someone bought a YubiKey.
|
||||||
|
|
||||||
|
> (Also, fun fact: the YubiKeys are technically limited, but there's over 1,000 of them. The more you know 🤷♂️)
|
||||||
|
|
||||||
|
In the future, I would definitely use libraries such as Zod to validate my scraper - it would have saved _so_ much time.
|
||||||
|
|
||||||
|
### How the scraper works
|
||||||
|
|
||||||
|
It's actually surprisingly simple! Under the hood, Hack Club's website uses Next.js, and Next.js includes a `script` tag with the ID of `__NEXT_DATA__` with a bunch of useful information. That useful information just so happens to contain the Arcade Shop's items in a nice and easy-to-use format.
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>View the scraper's code (~50 lines of Rust)</summary>
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use reqwest::Client;
|
||||||
|
use scraper::{Html, Selector};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use worker::*;
|
||||||
|
|
||||||
|
#[derive(Serialize, Deserialize, PartialEq, Eq, Default, Clone)]
|
||||||
|
pub struct ShopItem {
|
||||||
|
#[serde(rename = "Full Name")]
|
||||||
|
pub full_name: String,
|
||||||
|
|
||||||
|
#[serde(rename = "Description")]
|
||||||
|
pub description: Option<String>,
|
||||||
|
|
||||||
|
#[serde(rename = "Fulfillment Description")]
|
||||||
|
pub fulfillment_description: Option<String>,
|
||||||
|
|
||||||
|
#[serde(rename = "Cost Hours")]
|
||||||
|
pub price: i32,
|
||||||
|
|
||||||
|
#[serde(rename = "Stock")]
|
||||||
|
pub stock: Option<i32>,
|
||||||
|
|
||||||
|
pub id: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub type ShopItems = Vec<ShopItem>;
|
||||||
|
|
||||||
|
const USER_AGENT: &str = "Arcade-Monitor/1.0 (+@SkyfallWasTaken)";
|
||||||
|
|
||||||
|
pub async fn try_fetch(shop_url: Url) -> Result<ShopItems> {
|
||||||
|
let client = Client::new();
|
||||||
|
let response = client
|
||||||
|
.get(shop_url)
|
||||||
|
.header("User-Agent", USER_AGENT)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let doc_html = response.text().await.unwrap();
|
||||||
|
|
||||||
|
let doc = Html::parse_document(&doc_html);
|
||||||
|
let selector = Selector::parse("#__NEXT_DATA__").unwrap();
|
||||||
|
let json: serde_json::Value = serde_json::from_str(
|
||||||
|
&doc.select(&selector)
|
||||||
|
.next()
|
||||||
|
.ok_or("no #__NEXT_DATA__ element in document")?
|
||||||
|
.inner_html(),
|
||||||
|
)?;
|
||||||
|
|
||||||
|
let available_items = serde_json::from_value(
|
||||||
|
json.pointer("/props/pageProps/availableItems")
|
||||||
|
.ok_or("availableItems not found - is the ARCADE_SHOP_URL correct?")?
|
||||||
|
.clone(),
|
||||||
|
)?;
|
||||||
|
|
||||||
|
Ok(available_items)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>View the original scraper's code (30 lines of TypeScript)</summary>
|
||||||
|
<p>Note that the type definitions are incorrect.</p>
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { parse as parseHtml } from "node-html-parser";
|
||||||
|
|
||||||
|
export type ShopItem = {
|
||||||
|
name: string;
|
||||||
|
description: string;
|
||||||
|
fulfillmentDescription: string;
|
||||||
|
id: string;
|
||||||
|
imageUrl: string;
|
||||||
|
maxOrderQuantity: number;
|
||||||
|
price: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
export default function parseArcadeShopHtml(html: string): ShopItem[] {
|
||||||
|
const document = parseHtml(html);
|
||||||
|
const inner = document.getElementById("\_\_NEXT_DATA\_\_")?.text!;
|
||||||
|
|
||||||
|
const items = JSON.parse(inner).props.pageProps.availableItems.map((item: any) => {
|
||||||
|
return {
|
||||||
|
name: item\["Name"],
|
||||||
|
description: item\["Description"],
|
||||||
|
fulfillmentDescription: item\["Fulfillment Description"],
|
||||||
|
id: item.id,
|
||||||
|
imageUrl: item\["Image URL"],
|
||||||
|
maxOrderQuantity: item\["Max Order Quantity"],
|
||||||
|
price: item\["Cost Hours"],
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
return items;
|
||||||
|
```
|
||||||
|
</details>
|
||||||
|
|
||||||
|
## v2: the Rust version
|
||||||
|
|
||||||
|
Since my code was borked anyway, I decided that I might as well just rewrite the whole thing. I decided to rewrite it in Rust! Technically, I could've made the rewrite in TypeScript and it would've been fine, but I decided to rewrite it in Rust instead for a couple reasons:
|
||||||
|
|
||||||
|
- Serde is great and reduces boilerplate
|
||||||
|
- Rust's error handling is _superb._ I would absolutely love `Result<T, E>` to come to Go or TypeScript (and actually be used by the wider community)
|
||||||
|
- I just like Rust :D
|
||||||
|
|
||||||
|
The new version took a _lot_ less time to implement (around 3 hours) - in fact, it took less time to write than the initial version written in TypeScript! Most of my code after that was for things like enhancements and new features.
|
||||||
|
|
||||||
|
Three hours later... it worked!
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Since then, most of the changes have been things like the ticket/price ratio or prettier Slack messages that use Block Kit.
|
||||||
|
|
||||||
|
Overall, I'd call the Rust rewrite a huge success!
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_© 2024 Mahad Kalam_
|
||||||
|
|
||||||
|
_Licensed under the [MIT License](https://github.com/SkyfallWasTaken/arcade-monitor/blob/main/LICENSE.md)_
|
||||||
Loading…
Reference in a new issue