V3.1.0 (#158)

## Pull Request Template

### Prerequisites

<!-- Take a couple of minutes to help our maintainers work faster by
checking of the pre-requisites. -->
<!-- To tick the checkboxes replace the space with an 'x', so [ ]
becomes [x] . -->

- [x] I have
[searched](https://github.com/DefinetlyNotAI/Logicytics/pulls) for
duplicate or closed issues.
- [x] I have read the [contributing
guidelines](https://github.com/DefinetlyNotAI/Logicytics/blob/main/CONTRIBUTING.md).
- [x] I have followed the instructions in the
[wiki](https://github.com/DefinetlyNotAI/Logicytics/wiki) about
contributions.
- [x] I have updated the documentation accordingly, if required.
- [ ] I have tested my code with the `--dev` flag, if required.

### PR Type

<!-- Take a couple of minutes to help our maintainers work faster by
telling us what is the PR guided on. -->
<!-- To tick the checkboxes replace the space with an 'x', so [ ]
becomes [x] . -->

- [x] Bug fix <!-- Non-Breaking Bug Fix - Usually relates to fixing an
issue -->
- [x] New feature <!-- Non-Breaking Change that adds a new feature -->
- [x] Refactoring <!-- Non-Breaking Change that modifies existing code
to refactor it to become more organised -->
- [ ] Documentation
update <!-- Non-Breaking Change that modifies existing documentation to
refactor it or add extra comments - either wiki, md files or code is
included here -->
- [ ] ⚠️ Breaking change ⚠️ <!-- Breaking Bug Fix / New Addition that
changes how Logicytics works -->

### Description

<!-- REQUIRED: Provide a summary of the PR and what you expected to
happen. -->
CodeRabbit will supply this

### Motivation and Context

<!-- REQUIRED: Why is this PR required? What problem does it solve? Why
do you want to do it? -->
To maintain v3
### Credit

<!-- If this PR is a contribution, please mention the contributors here
using the appropriate syntax. -->

<!--
### File-Created/CONTRIBUTION by MAIN-Username
What you did, created, removed, refactored, fixed, or discovered.
- [Your GitHub Username](https://github.com/YourGitHubLink)
- [Your GitHub Username](https://github.com/YourGitHubLink) etc...
-->
_N/A_

### Issues Fixed

<!-- REQUIRED: What issues will be fixed? (Format: "#50, #23" etc.) if
none exist type _N/A_ -->
_N/A_

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced a framework for visualizing neural network models and their
features.
	- Added functionality to capture and log system memory information.
- Enhanced scanning functionality for files and directories for
sensitive content.
- Added new entries to the `.gitignore` file for improved file
management.
- Implemented a global locking mechanism for thread safety during model
and vectorizer usage.
	- Added a new section in the configuration file for study parameters.

- **Bug Fixes**
	- Improved error handling and logging in various scripts.

- **Documentation**
- Updated documentation for VulnScan to include new repository details
and resources.
	- Enhanced the `README.md` for better clarity and usability.

- **Chores**
- Updated configuration file to reflect new settings for model and
vectorizer paths.
- Modified `requirements.txt` to add and remove several package
dependencies.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

.gitignore

@@ -319,3 +319,4 @@ $RECYCLE.BIN/
 *.pyc
 /CODE/SysInternal_Suite/.sys.ignore
 /ACCESS/
+/CODE/VulnScan/tools/NN features/

CODE/Logicytics.py

@@ -16,11 +16,12 @@
 # Initialization
 FileManagement.mkdir()
 log = Log({"log_level": DEBUG, "delete_log": DELETE_LOGS})
+ACTION = None
+SUB_ACTION = None
 
 
 class Health:
     @staticmethod
-    @log.function
     def backup(directory: str, name: str):
         """
         Creates a backup of a specified directory by zipping its contents and moving it to a designated backup location.
@@ -47,7 +48,6 @@ def backup(directory: str, name: str):
         shutil.move(f"{name}.zip", "../ACCESS/BACKUP")
 
     @staticmethod
-    @log.function
     def update() -> tuple[str, str]:
         """
         Updates the repository by pulling the latest changes from the remote repository.
@@ -325,9 +325,13 @@ def threaded_execution(execution_list_thread, index_thread):
 
 def zip_generated_files():
     """Zips generated files based on the action."""
-
-    def zip_and_log(directory, name):
-        zip_values = FileManagement.Zip.and_hash(directory, name, ACTION)
+    def zip_and_log(directory: str, name: str):
+        log.debug(f"Zipping directory '{directory}' with name '{name}' under action '{ACTION}'")
+        zip_values = FileManagement.Zip.and_hash(
+                directory,
+                name,
+                ACTION if ACTION is not None else f"ERROR_NO_ACTION_SPECIFIED_{datetime.now().isoformat()}"
+        )
         if isinstance(zip_values, str):
             log.error(zip_values)
         else:

CODE/VulnScan/Documentation.md

@@ -107,3 +107,32 @@ VulnScan is designed to detect sensitive data across various file formats. It of
 - **Progress Tracking**: Visualizes accuracy and loss per epoch with graphs.
 - **Error Handling**: Logs errors for missing files, attribute issues, or unexpected conditions.
 - **Extensibility**: Supports plug-and-play integration for new algorithms or datasets.
+
+
+# More files
+
+There is a repository that archived all the data used to make the model, 
+as well as previously trained models for you to test out 
+(loading scripts and vectorizers are not included). 
+
+The repository is located [here](https://github.com/DefinetlyNotAI/VulnScan_TrainingData).
+
+The repository contains the following directories:
+- `Training Data`: Contains the data used to train the models. Is organized by the file size and amount, unless its Tests, where they explicitly say text.
+- `Archived Models`: Contains the previously trained models. Is organized by the model type then version.
+- `NN features`: Contains information about the model `.3n3` and the vectorizer used. Information include:
+  - `Documentation_Study_Network.md`: A markdown file that contains more info.
+  - `Neural Network Nodes Graph.gexf`: A Gephi file that contains the model nodes and edges.
+    - `Nodes and edges (GEPHI).csv`: A CSV file that contains the model nodes and edges.
+    - `Statistics`: Directories made by Gephi, containing the statistics of the model nodes and edges.
+  - `Feature_Importance.svg`: A SVG file that contains the feature importance of the model.
+  - `Loss_Landscape_3D.html`: A HTML file that contains the 3D loss landscape of the model.
+  - `Model Accuracy Over Epochs.png` and `Model Loss Over Epochs.png`: PNG files that contain the model accuracy and loss over epochs.
+  - `Model state dictionary.txt`: A text file that contains the model state dictionary.
+  - `Model Summary.txt`: A text file that contains the model summary.
+  - `Model Visualization.png`: A PNG file that contains the model visualization.
+  - `Top_90_Features.svg`: A SVG file that contains the top 90 features of the model.
+  - `Vectorizer features.txt`: A text file that contains the vectorizer features.
+  - `Visualize Activation.png`: A PNG file that contains the visualization of the model activation.
+  - `Visualize t-SNE.png`: A PNG file that contains the visualization of the model t-SNE.
+  - `Weight Distribution.png`: A PNG file that contains the weight distribution of the model.