diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..a865d6d --- /dev/null +++ b/docs/README.md @@ -0,0 +1,35 @@ +# Documentation + +## File Formats + +Documentation and examples of file formats. + +### [Legacy Manifest](file_formats/LEGACY_MANIFEST.md) + +Blue Hell Productions `manifest.xml`. Still in use by W3D Hub. + +### [Megafest](file_formats/MEGAFEST.md) + +Prototype new thing `megafest.json` + +### [MIX](file_formats/MIX.md) + +Westwood archive format `.mix`, `.dat` and `.pkg` + +### [W3D Hub Patch](file_formats/W3D_HUB_PATCH.md) + +Describes how to update a MIX archive from a package patch + +## Protocols + +### [LAN Discovery](protocols/LAN_DISCOVERY.md) + +How launchers and other applications discovery each other. + +### [Launcher Remote](protocols/LAUNCHER_REMOTE.md) + +Remotely control and get progress reports from the launcher + +### [LAN Package Share](protocols/LAN_PACKAGE_SHARE.md) + +Enable launchers on the same network to discover each other and share packages over LAN. diff --git a/docs/file_formats/LEGACY_MANIFEST.md b/docs/file_formats/LEGACY_MANIFEST.md new file mode 100644 index 0000000..22ebe82 --- /dev/null +++ b/docs/file_formats/LEGACY_MANIFEST.md @@ -0,0 +1,73 @@ +# Legacy Manifest + +In use since Blue Hell Productions + +> [!IMPORTANT] +> References to packages exclude `.zip` + +Example `Full` manifest: + +```xml + + + + + + + + + + + + + + + + +``` + +Example `Patch` manifest: + +```xml + + + + + + + + + + + + + + + + + + + +``` + +## SPECIFICATION +### BHP_GAME_MANIFEST (Root Element) +* **GAME** - Unique Application ID +* **VERSION** - Application Unique Version for Build +* **TYPE** - Type of Build + * **Full** - Full Build + * **Patch** - Patch Build +* **BASEVERSION** - *OPTIONAL* Version this build was based upon + +### DEPENDENCY +* **NAME** - Package name of dependency sans file extension +* **REMOVEDSINCE** - _OPTIONAL_ No data. Not used anymore? + +### FILE +* **NAME** - Name of file +* **CHECKSUM** - _OPTIONAL_ SHA256 hash of file +* **PACKAGE** - _OPTIONAL_ Package that contains this file. Not present on removed files or files that have been patched. +* **REMOVEDSINCE** - _OPTIONAL_ Version this package was removed in. If present, only _NAME_ and _REMOVEDSINCE_ will be present. + * **PATCH** + * **FROM** - Version that the file last changed + * **PACKAGE** - Name of package that contains this file. \ No newline at end of file diff --git a/docs/file_formats/MEGAFEST.md b/docs/file_formats/MEGAFEST.md new file mode 100644 index 0000000..959c9a0 --- /dev/null +++ b/docs/file_formats/MEGAFEST.md @@ -0,0 +1,319 @@ +> [!NOTE] +> DRAFT + +# Megafest + +The fat manifest for W3D Hub Packager and Launcher(s), represented as a JSON hash. + +This **MEGAFEST** is intended to simplify patch generation and distribution by ensuring that +all the data the Packager and Launcher(s) need can be found in a **single** manifest file. + +Speed up application packaging by only needing to scan and checksum the new builds files instead of needing +to maintain a pristine copy of the previous version to be diffed against. + +Reduce the number of requests that the launcher(s) need to make to sort out how to download and +patch the application. + +## Glossary + +### File + +An individual file on disk. + +An individual file as part of a MIX archive. + +### Package + +A collection of one or more files in a compressed archive for efficiently distributing the application. + +> [!NOTE] +> Packages _MAY_ be compressed using ZSTANDARD (.zst) in the future. Do not assume all packages are zip files. +> +> Package names in the megafest include the file extension for this reason. + +## Example `megafest.json` File + +```json +{ + "spec": 0, + "application": { + "id": "apb", + "version": "3.7.0.1", + "user_level": "public", + "previous_versions": [ + "3.7.0.0" + ] + }, + "dependencies": [ + "msvc-2022" + ], + "packages": [ + { + "name": "binaries.zip", + "version": "3.7.0.0", + "checksum": "SHA256-HASH", + "size": 4096, + "chunk_size": 4194304, + "chunk_checksums": [ + "SHA256-HASH-A", + "SHA256-HASH-B" + ], + "files": [ + { + "name": "game2.exe", + "checksum": "SHA256-HASH", + "size": 8691743 + } + ] + }, + { + "name": "data/Always.dat.zip", + "version": "3.7.0.0", + "checksum": "SHA256-HASH", + "size": 1355917483, + "chunk_size": 4194304, + "chunk_checksums": [ + "SHA256-HASH-A", + "SHA256-HASH-B" + ], + "files": [ + { + "name": "mp_wep_gdi.w3d", + "checksum": "CRC32-HASH", + "offset": 0, + "size": 8691743 + } + ] + }, + { + "name": "data/Always.patch.3.7.0.0.zst", + "version": "3.7.0.1", + "checksum": "SHA256-HASH", + "size": 1355919483, + "chunk_size": 4194304, + "chunk_checksums": [ + "SHA256-HASH-A", + "SHA256-HASH-B" + ], + "from_version": "3.7.0.0", + "files": [ + { + "name": "data/Always.patch", + "checksum": "SHA256-HASH", + "size": 8691743 + } + ] + }, + { + "name": "binaries.zip", + "version": "3.7.0.1", + "checksum": "SHA256-HASH", + "size": 4096, + "chunk_size": 4194304, + "chunk_checksums": [ + "SHA256-HASH-A", + "SHA256-HASH-B" + ], + "files": [ + { + "name": "game.exe", + "checksum": "SHA256-HASH", + "size": 8691743 + } + ] + } + ], + "changes": [ + { + "name": "game.exe", + "type": "added", + "package": "binaries" + }, + { + "name": "game2.exe", + "type": "removed" + }, + { + "name": "data/Always.dat", + "type": "updated", + "package": "Always.dat" + } + ], + "index": [ + { + "name": "game.exe", + "checksum": "SHA256-HASH", + "size": 8691743 + }, + { + "name": "data/Always.dat", + "checksum": "SHA256-HASH", + "size": 1355917483 + } + ] +} +``` + +## SPECIFICATION + +### Application + +List of application details: + +* **ID** - Unique Application ID +* **VERSION** - Application Unique Version +* **USER_LEVEL** - User access level for this version +* **PREVIOUS_VERSIONS** - Complete list of previous versions. Full builds only include _the_ previous version + +Example: + +```json +{ + "id": "apb", + "version": "3.7.0.1", + "user_level": "public", + "previous_versions": [ + "3.7.0.0" + ] +} +``` + +### Dependencies + +List of application's dependencies + +Example: + +```json +[ + "msvc-2022" +] +``` + +### Packages + +Complete list of application's version tree of packages. + +That is, every package from the last full build to present for this version series. + +* **NAME** - Name of package +* **VERSION** - Application version this package belongs to +* **CHECKSUM** - SHA256 checksum of package +* **SIZE** - File size of package in bytes +* CHUNK **SIZE** - Size _CHUNK_CHECKSUMS_ represent in bytes +* **CHUNK_CHECKSUMS** - Array of file chunk checksums +* **FROM_VERSION** - *OPTIONAL* Version this file was last changed. Only present for MIX patches. +* **FILES** - Array of files + * **NAME** + * **CHECKSUM** - SHA256 checksum of file. + * **SIZE** - File size in bytes + +Example: + +```json +[ + { + "name": "binaries", + "version": "3.7.0.0", + "checksum": "SHA256-HASH", + "size": 4096, + "chunk_size": 4194304, + "chunk_checksums": [ + "SHA256-HASH-A", + "SHA256-HASH-B" + ], + "files": [ + { + "name": "game.exe", + "checksum": "SHA256-HASH", + "size": 8691743 + } + ] + }, + { + "name": "data/Always.patch.3.7.0.0", + "version": "3.7.0.1", + "checksum": "SHA256-HASH", + "size": 2436543, + "chunk_size": 4194304, + "chunk_checksums": [ + "SHA256-HASH-A", + "SHA256-HASH-B" + ], + "from_version": "3.7.0.0", + "files": [ + { + "name": "data/Always.patch", + "checksum": "SHA256-HASH", + "size": 3636749 + } + ] + } +] +``` + +### Changes + +List of file changes in this build. + +* **NAME** - Name of file +* **TYPE** - Type of change + * **ADDED** - New file has been added + * **UPDATED** - File has been changed + * **REMOVED** - File has been deleted +* **PACKAGE** - _OPTIONAL_ Name of package, for this build version, the file can be found in + +```json +[ + { + "name": "game.exe", + "type": "updated", + "package": "binaries.zip" + }, + { + "name": "game2.exe", + "type": "removed" + }, + { + "name": "data/Always.dat", + "type": "added", + "package": "Always.dat.zip" + } +] +``` + +### Index + +Complete list of files for this build with: filename, checksum, file size, and list of files for MIX archives. + +* **NAME** - File name +* **CHECKSUM** - SHA256 hash of file +* **SIZE** - File size +* **FILES** - _OPTIONAL_ List of files inside of MIX archives + * **NAME** - File name + * **CHECKSUM** - CRC32 hash of file data + * **OFFSET** - Offset of file in MIX archive + * **SIZE** - Size of file data + +```json +[ + { + "name": "game.exe", + "checksum": "SHA256-HASH", + "size": 8691743 + }, + { + "name": "data/Always.dat", + "checksum": "SHA256-HASH", + "size": 570961432, + "files": [ + { + "name": "mp_wep_gdi.w3d", + "checksum": "CRC32_HASH", + "offset": 0, + "size": 3524 + } + ] + } +] +``` diff --git a/docs/file_formats/MIX.md b/docs/file_formats/MIX.md new file mode 100644 index 0000000..1a40575 --- /dev/null +++ b/docs/file_formats/MIX.md @@ -0,0 +1,132 @@ +# Westwood Renegade MIX Archive + +with W3D Hub / Tiberian Technologies alterations + +The .MIX archive file format is used by Renegade as _yes_. + +> [!IMPORTANT] +> The MIX format has no notion about directories. +> +> Each file name **MUST** be unique. +> +> e.g. `data/map.dds` and `data/RA_Under/MAP.dds` have the same base file name +> and therefore **MUST** raise an error when writing. + +> [!NOTE] +> `io_pos` is the current position on the seek head of the open file + +## Reading + +* Read [Header](#header) + * validate MIME type (`MIX1` or `MIX2`) +* Jump to io_pos `file_data_offset` +* Read `file_count` +* Read array of [`file_data`](#file-data-array) +* Jump to io_pos `file_names_offset` +* Read `file_count` +* Read array of [`file_names`](#file-names-array) +* Read file blobs + * For each file's [`file_data`](#file-data-array) + * Jump io_pos to `file_content_offset` + * Read `file_content_length` +* Done. + +## Writing + +* Write MIME type (`MIX1` or `MIX2`) +* Jump to io_pos `16` +* Write file blobs + * **IMPORTANT:** Add `-io_pos & 7` padding to io_pos **AFTER** each file blob is written + * Save io_pos as `file_data_offset` +* Write `file_count` + * Write array of [`file_data`](#file-data-array) + * Save io_pos as `file_names_offset` +* Write `file_count` + * Write array of [`file_names`](#file-names-array) + * **IMPORTANT:** Ensure file names are null terminated and DO NOT exceed 254 (+ null byte) characters in length. +* Jump to io_pos `4` + * Write [`file_data_offset`](#header) + * Write [`file_names_offset`](#header) + * Write [`reserved`](#header) +* Done. + +## Pseudo Example: + +```yml +header: + mime: MIX1 or MIX2 + file_data_offset: 0x1024EAEA + file_names_offset: 0x8192EAEA + reserved: 0 + +files: + array: + - file_blob + +file_data: + file_count: 0x000021 + array: + file_name_crc32: 0xEFEFEFEF + file_content_offset: 0x3217DEAD + file_content_length: 0x0018BEEF + +file_names: + file_count: 0x000021 + array: + file_name_length: 0xff + file_name: "mp_wep_gdi.w3d" +``` + +## MIME Types + +### MIX1 `0x3158494D` + +Westwood Farm Fresh Organic. + +### MIX2 `0x3258494D` + +Same as `MIX1`. `MIX2` hints to the engine's file reader to decrypt files before use. + +## Data Structures + +### Header + +| Name | Offset | Type | Description | +|-------------------|--------|---------|---------------------------------------------------------------------| +| MIME | 0 | int32_t | 4 bytes representing `MIX1` (`0x3158494D)` or `MIX2` (`0x3258494D`) | +| File Data Offset | 4 | int32_t | Offset in MIX archive that [`file_data`](#file-data) data starts | +| File Names Offset | 8 | int32_t | Offset in MIX archive that [`file_name`](#file-names) data starts | +| RESERVED | 12 | int32_t | Unused reserved int. Write as `0` | + +### File + +| Name | Offset | Type | Description | +|-----------|---------|-----------------------------|---------------| +| File Blob | complex | char[`file_content_length`] | File contents | + +### File Data + +| Name | Offset | Type | Description | +|------------|--------------------|---------|--------------------------------| +| File Count | `file_data_offset` | int32_t | Number of files in MIX archive | + +### File Data Array + +| Name | Offset | Type | Description | +|---------------------|--------------------------------------------|----------|--------------------------------------------------------| +| File Name CRC32 | `file_data_offset` + 4 * (index * sizeof) | uint32_t | CRC32 of **UPPERCASE** file name in network byte order | +| File Content Offset | `file_data_offset` + 8 * (index * sizeof) | uint32_t | File content offset | +| File Content Length | `file_data_offset` + 12 * (index * sizeof) | uint32_t | File content length | + +### File Names + +| Name | Offset | Type | Description | +|------------|---------------------|---------|--------------------------------| +| File Count | `file_names_offset` | int32_t | Number of files in MIX archive | + +### File Names Array + +| Name | Offset | Type | Description | +|------------------|---------|--------------------------|-------------------------------------------------| +| File Name Length | complex | uint8_t | Length of string in bytes, including null byte. | +| File Name | complex | char[`file_name_length`] | null terminated string | diff --git a/docs/file_formats/W3D_HUB_PATCH.md b/docs/file_formats/W3D_HUB_PATCH.md new file mode 100644 index 0000000..b1eee8d --- /dev/null +++ b/docs/file_formats/W3D_HUB_PATCH.md @@ -0,0 +1,19 @@ +# W3D Hub Patch +A W3D Hub Patch + +A `.w3dhub.patch` is included in MIX Patch files. +It contains instructions for updating the target MIX. + +JSON file with a static CRC32 of `E6FE46B8` + +## Example: +```json +{ + "removedFiles": [ + "sov_v_mad.gif" + ], + "updatedFiles": [ + "sov_v_mad.w3d" + ] +} +``` diff --git a/docs/protocols/LAN_DISCOVERY.md b/docs/protocols/LAN_DISCOVERY.md new file mode 100644 index 0000000..957da25 --- /dev/null +++ b/docs/protocols/LAN_DISCOVERY.md @@ -0,0 +1,36 @@ +> [!NOTE] +> DRAFT + +# LAN Discovery + +Doing things + +Broadcast port: 4898 + +| Name | Type | Description | +|-------------------|----------|--------------------------------------------------------------------------| +| Version | int32_t | Version of protocol this application supports | +| Owner | string | Nickname of application's active user | +| Hostname | string | Name of device | +| UUID | string | Unique identifier for this application | +| Application | string | Name of application | +| Features | array | Array of strings naming features the application supports | +| Service Port | uint16_t | Dynamically assigned TCP port that interested parties should connect too | + +> [!NOTE] +> Max packet size for UDP broadcast may need to be limited to 512 bytes + +```json +{ + "version": 0, + "owner": "cyberarm", + "hostname": "PC-1692", + "uuid": "019bcf1e-a22e-7fe0-a3db-3a9d37bfc6fa", + "application": "W3D Hub Linux Launcher", + "features": [ + "launcher_remote:3", + "package_share:1" + ], + "service_port": 56802 +} +``` diff --git a/docs/protocols/LAN_PACKAGE_SHARE.md b/docs/protocols/LAN_PACKAGE_SHARE.md new file mode 100644 index 0000000..feac4a4 --- /dev/null +++ b/docs/protocols/LAN_PACKAGE_SHARE.md @@ -0,0 +1,9 @@ +> [!NOTE] +> DRAFT + +# LAN Package Share + +Authentication is not required, only user opt-in to enable package sharing from their application. + +* List available packages +* upload or download verified packages between other applications on LAN network diff --git a/docs/protocols/LAUNCHER_REMOTE.md b/docs/protocols/LAUNCHER_REMOTE.md new file mode 100644 index 0000000..4172801 --- /dev/null +++ b/docs/protocols/LAUNCHER_REMOTE.md @@ -0,0 +1,9 @@ +> [!NOTE] +> DRAFT + +# Launcher Remote + +* Get list of available, installed, and in progress applications +* Trigger download or update of application +* Delete application +* Join server