Feat: Add WebSockets, graph generation service and HA

[Its4Nik]

What's changed?

• New unit tests
• Optimized CI/CD workflow(s)
• Websocket connections
• Dockerfile changes
• A bunch of new npm run commands
• Adjusted weird container percentage calculation I did (seriously what did i do therre?!)
• A graph generation "service" (Needs a rewrite)
• High Availability
• And more!

Note

This took a bit longer then expected but hey atleast the Frontend gets some love as well right now (See this issue: #56)

Summary by Sourcery

Remove Swagger annotations from routes and update documentation.

New Features:

• Add WebSocket endpoints for streaming container data and server logs.
• Introduce stack management routes for creating, starting, stopping, and retrieving stack configurations and environment variables.
• Add graph generation functionality with routes for retrieving graph data in JSON and HTML formats, as well as a PNG image of the graph.

Enhancements:

• Remove Swagger annotations from route files.
• Update container data fetching to include CPU usage as a percentage.
• Improve error handling in container data fetching.
• Refactor frontend configuration controllers for better error handling.
• Update high availability controllers for improved error handling and file management.
• Improve logging and error handling in configuration handlers.
• Update notification template handling to improve error handling and data parsing.
• Refactor Docker client retrieval and add debug logging.
• Update logger to filter exit listener logs and improve formatting.
• Improve atomic write utility to handle various data types.

Documentation:

• Update documentation to reflect the removal of Swagger annotations and other changes.

Tests:

• Add tests for frontend configuration, API getters, database interactions, configuration endpoints, and authentication.

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR implements a broad set of improvements and new features across the codebase. The changes include the addition of extensive unit tests for various modules, optimizations to the CI/CD workflows, implementation of WebSocket connectivity for container data and log streaming, a new graph generation service, significant updates to High Availability handling, and refinements in the frontend configuration endpoints. Additionally, the PR refactors error handling/logging, updates Docker and npm run commands, and overhauls the Swagger documentation by moving configuration from code annotations to an external YAML file.

No diagrams generated as the changes look simple and do not need a visual representation.

File-Level Changes

Change	Details	Files
Added new unit tests across the codebase	Introduced tests for authentication, frontend configurations, getters, database queries, config endpoints, and stack management Created new test files under the tests directory (e.g., auth.spec.ts, database.spec.ts, frontend.spec.ts, getters.spec.ts, config.spec.ts, stacks tests)	__tests__/auth.spec.ts __tests__/database.spec.ts __tests__/frontend.spec.ts __tests__/getters.spec.ts __tests__/config.spec.ts __tests__/util/previousResponse.ts
Optimized CI/CD workflows and build configurations	Optimized GitHub Actions workflows (validation.yaml, build-image.yaml) for better performance and additional steps such as automated TODO issue creation Updated package.json scripts with new run commands (test, dev:socket, build:docker:prod, etc.) Made modifications in entrypoint.sh for environment management	.github/workflows/validation.yaml docker/docker-compose.yaml docker/Dockerfile-base docker/Dockerfile-dev package.json src/misc/createEnvDev.sh src/misc/createEnvFile.sh src/misc/entrypoint.sh
Implemented WebSocket connections for container data and log streaming	Added a new module (src/utils/webSocket.ts) to set up and handle WebSocket connections Updated init.ts to initialize WebSocket server on upgrade events	src/utils/webSocket.ts src/init.ts
Introduced a graph generation service for visualizing container deployments	Implemented the graph generation logic in a new handler (src/handlers/graph.ts) using Cytoscape and Puppeteer Added new routes to serve the generated graph HTML and image (src/routes/graphs/routes.ts) Integrated asset files for icons used in the graph visualization	src/handlers/graph.ts src/routes/graphs/routes.ts src/utils/assets/server-icon.svg src/utils/assets/container-icon.svg src/utils/assets/api-icon.svg
Enhanced High Availability and configuration handling	Refactored high availability functions to use improved error logging and atomic file writes (src/controllers/highAvailability.ts) Modified host system configuration logging and file generation (src/config/hostsystem.ts) Updated configuration management scripts to create and update necessary files	src/controllers/highAvailability.ts src/config/hostsystem.ts src/config/initFiles.ts
Updated and refactored frontend configuration endpoints	Refactored routes and controllers for frontend configuration (hide, show, add tag, remove tag, pin/unpin, add/remove link/icon) Improved error handling and logging in frontendConfiguration.ts	src/routes/frontendController/routes.ts src/controllers/frontendConfiguration.ts src/data/frontendConfiguration.json
Updated Docker configurations and npm run commands	Revised Dockerfile and docker-compose setups for both production and development environments Added new commands and optimized existing scripts in package.json	package.json docker/Dockerfile-base docker/Dockerfile-dev docker/docker-compose.yaml docker/docker-compose.dev.yaml
General refactoring and improved error handling/logging	Standardized error message extraction using error instance checks across controllers and utils Refined logger formatting and removed redundant console.error invocations Updated utility modules (atomicWrite, dockerClient, containerService, etc.) to improve stability	src/utils/logger.ts src/utils/atomicWrite.ts src/utils/dockerClient.ts src/utils/containerService.ts src/handlers/response.ts src/controllers/fetchData.ts src/handlers/notification.ts src/controllers/scheduler.ts src/controllers/proxy.ts
Revamped Swagger API documentation integration	Migrated from inline JSDoc swagger comments to an external swagger.yaml file Updated swagger configuration options in src/config/swaggerConfig.ts and related files Removed deprecated swagger comments from route files	src/config/swaggerConfig.ts src/config/swagger.yaml src/utils/swaggerDocs.ts
Introduced new stack management functionality	Added backend support for docker stack operations (create, start, stop, get, set-env, get-env) via new handler (src/handlers/stack.ts) Created new routes for stack management (src/routes/stack/routes.ts) Defined new types for Docker compose and stack configuration (src/typings/dockerCompose.ts, src/typings/stackConfig.ts, src/typings/dockerStackEnv.ts)	src/handlers/stack.ts src/routes/stack/routes.ts src/typings/dockerCompose.ts src/typings/stackConfig.ts src/typings/dockerStackEnv.ts

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!
• Generate a plan of action for an issue: Comment @sourcery-ai plan on
  an issue to generate a plan of action for it.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.15 s (691.2 files/s, 39149.2 lines/s)

Language	files	blank	comment	code
TypeScript	79	678	52	4098
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	808	198	4941

[sourcery-ai]

Hey @Its4Nik - I've reviewed your changes - here's some feedback:

Overall Comments:

• Consider extracting the error conversion logic (error instanceof Error ? error.message : String(error)) into a helper to reduce duplication across files.
• When launching puppeteer in renderGraphToImage, validate the presence of required system dependencies earlier to provide clearer failure messages.

Here's what I looked at during the review

• 🟡 General issues: 5 issues found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

question (bug_risk): Return type consistency in error handling.

Previously the function returned an object on error; now it returns false. Ensure that the change in return type is intended and that the caller handles the different type correctly.

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.15 s (699.1 files/s, 39594.4 lines/s)

Language	files	blank	comment	code
TypeScript	79	678	52	4098
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	808	198	4941

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.15 s (700.5 files/s, 39677.0 lines/s)

Language	files	blank	comment	code
TypeScript	79	678	52	4098
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	808	198	4941

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.14 s (737.2 files/s, 41752.4 lines/s)

Language	files	blank	comment	code
TypeScript	79	678	52	4098
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	808	198	4941

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.15 s (678.1 files/s, 38407.6 lines/s)

Language	files	blank	comment	code
TypeScript	79	679	52	4097
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	809	198	4940

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.15 s (698.0 files/s, 39535.2 lines/s)

Language	files	blank	comment	code
TypeScript	79	679	52	4097
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	809	198	4940

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.16 s (649.6 files/s, 36793.2 lines/s)

Language	files	blank	comment	code
TypeScript	79	679	52	4097
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	809	198	4940

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.16 s (656.6 files/s, 37191.1 lines/s)

Language	files	blank	comment	code
TypeScript	79	679	52	4097
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	809	198	4940

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.15 s (688.1 files/s, 39103.5 lines/s)

Language	files	blank	comment	code
TypeScript	79	681	51	4116
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	811	197	4959

[github-actions]

cloc	github.com/AlDanial/cloc v 1.96 T=0.15 s (699.3 files/s, 39741.1 lines/s)

Language	files	blank	comment	code
TypeScript	79	681	51	4116
Text	10	9	0	275
JavaScript	2	25	136	210
Bourne Shell	8	55	10	204
Markdown	3	41	0	151
SVG	3	0	0	3
--------	--------	--------	--------	--------
SUM:	105	811	197	4959

[Its4Nik]

Needs a rewrite

[Its4Nik]

No bigger issues in my eyes