Azure Blob Storage, Azure File Share and SharePoint Online Connector Apps #23225

IceOnly · 2023-05-05T10:37:18Z

This PR contains three new connector apps, to connect Azure Blob Storage, Azure File Share and SharePoint Online with the New File System Module in the System App.

File System Module PR:
microsoft/BCApps#663

Here are some Screenshots:

Fixes #22691
Fixes AB#559148

Apps/W1/File - Azure BLOB Storage Connector/app/src/BlobStorageAccount.Page.al

Apps/W1/File - Azure BLOB Storage Connector/app/src/BlobStorageConnectorImpl.Codeunit.al

Modules/System/File Access/src/Connector/FileConnector.Interface.al

pri-kise · 2023-05-05T22:53:20Z

Some general thoughts:
I think we need some kind of path serparated from the name added to the file account content.
It's often very important to know the path.

Additionally we might need some additional kind of top level selection before List Directories:
I'm thinking about a webservice for local file system access, there we should be able to select the Drive "C:" or "D:".
For Azure File Share we should be able to select the File Share on a top level before searching directories and files.

Further more if I take a look at the current sharepoint client. I have no idea how we should map this to the current provided
interface "File Connector".
https://github.com/microsoft/ALAppExtensions/blob/main/Modules/System/SharePoint/src/SharePointClient.Codeunit.al

Sometimes a path isn't enough and we might need more information to download a file.
Maybe this info could be added to the table 70005 "File Account Content" table and the interface "File Connector" could accept a "File Account Content" as parameter instead of a path....

Maybe someone else with more experience has some ideas how this access can be unified...

IceOnly · 2023-05-06T17:08:13Z

Some general thoughts: I think we need some kind of path serparated from the name added to the file account content. It's often very important to know the path.

Additionally we might need some additional kind of top level selection before List Directories: I'm thinking about a webservice for local file system access, there we should be able to select the Drive "C:" or "D:". For Azure File Share we should be able to select the File Share on a top level before searching directories and files.

Further more if I take a look at the current sharepoint client. I have no idea how we should map this to the current provided interface "File Connector". https://github.com/microsoft/ALAppExtensions/blob/main/Modules/System/SharePoint/src/SharePointClient.Codeunit.al

Sometimes a path isn't enough and we might need more information to download a file. Maybe this info could be added to the table 70005 "File Account Content" table and the interface "File Connector" could accept a "File Account Content" as parameter instead of a path....

Maybe someone else with more experience has some ideas how this access can be unified...

Why not listing C: and D: as directory on the top level?
The path is part of the Fiel Acount Content table. It is splitted in Parent Folder and Name.

However, I am not sure if the module should define the path separator. So all Services uses the same path structure and the implentation need to translate if needed.

Apps/W1/File - Azure BLOB Storage Connector/app/src/BlobStorageConnectorImpl.Codeunit.al

PeterConijn

Looks cool! I have a few small performance and security concerns, which I have addressed, and some minor other stuff, but nothing major. I'm excited for this.

Apps/W1/File - Azure BLOB Storage Connector/app/src/BlobStorageAccount.Table.al

Apps/W1/File - Azure BLOB Storage Connector/app/src/BlobStorageAccountWizard.Page.al

Modules/System/File Access/src/Account/FileAccountImpl.Codeunit.al

JesperSchulz · 2023-08-18T07:29:04Z

Let me amplify this on Twitter/X to get your initial question answered: "before I go round the whole thing, I'm interested in whether there is any interest in the module at all". Let's see what people have to say! 😊

tinfister · 2023-08-18T07:40:00Z

I support every effort to improve file handling.

dNsl9r · 2023-08-18T11:43:44Z

Wonderful! Please merge this 👍🏻

miljance · 2023-08-18T15:21:28Z

Easily overlooked feature but a needed one. Huge development effort that should not be thrown away.
Perhaps I will find some time to invest in contribution from my side but cannot make promises at the moment.

TheDoubleH · 2023-08-19T07:09:14Z

Bravo! Love seeing more 'helper' features/functions that eventually will benefit all of us!

JesperSchulz · 2023-10-06T06:39:55Z

@IceOnly, this PR seems to have stagnated a little (which is perfectly fine). I was just wondering if you want to push this forward at some point, or if you've changed your mind / cannot find the time? There seems to be plenty of interest for this module in the community!

IceOnly · 2023-10-06T08:07:24Z

@IceOnly, this PR seems to have stagnated a little (which is perfectly fine). I was just wondering if you want to push this forward at some point, or if you've changed your mind / cannot find the time? There seems to be plenty of interest for this module in the community!

It is the time. I will try to find some free time to make the last changes.

This are my open points:

I am struggling with the wist from pri-kise to split the Main Interface in logic interfaces like one for File, one for Directory and one for Account access. I like the Idea to implement one Interface and you are done.
I will try to implement a realy simple pagination interface. The Most file services uses realy different methods for getting the next batch. So i will only alow the implemntation to result a pagination codeunit that can store the inforamtion it needs to get the next batch. How big a batch is can only be defined by the implementation.

JesperSchulz · 2023-10-06T10:18:53Z

Feel free to do it at your own pace. I didn't mean to stress you. I was just curious if you still intend to continue work. If you do, we'll just leave this PR open. No problem whatsoever.

…SAS procedure to use Secret directly

…ageConnectorImpl to use Authorization Type directly

…age across multiple files

…d CreateAccount procedures to use temporary variable names for clarity

Migration of microsoft/ALAppExtensions#23225 Now that we have made several clients Universal code ready. One of the main problems turned out to be the replacement of the file record and the file object (File.Create, File.Exists). Some of our customers now use Azure Blob Service from the System app. Others don't have a good internet and now use a local microservice with REST API, others use Azure File Shares. The problem is that every integration is different. I think many other partners will have the same problem. To simplify access I have adopted the email module and created new file accounts. With just one codeunit, a developer can now connect to various file services without having to know how they actually work. The code unit "File System" delviers everything taht is needed. An additional PR contains three connector apps: microsoft/ALAppExtensions#23225 This Apps will conenct: - Azure Blob Storage - Azure File Share - SharePoint Online All three service can be access over **one** unified interface! New provider in the future could be: - a OneDrive Provider - Third Party file services An example that shows how simple it is to use the new modules can be found here: https://github.com/IceOnly/BC_FileSystem_Example But before I go round the whole thing, I'm interested in whether there is any interest in the module at all. Here are some Screenshots: ![image](https://user-images.githubusercontent.com/3911556/236433843-11ee0b26-ee9c-4da2-8bc6-efb32675090c.png) ![image](https://user-images.githubusercontent.com/3911556/236433887-bf2bf2f2-f68e-4efa-9db5-074a6f68fad0.png) ![image](https://user-images.githubusercontent.com/3911556/236433950-e8117496-16f6-4b25-a575-42ba729ca6c3.png) ![image](https://user-images.githubusercontent.com/3911556/236434182-97fd304a-95ac-4180-9d52-9ead78899822.png) ![image](https://user-images.githubusercontent.com/3911556/236435173-a01a77bc-04de-4063-b0bb-bf18e28b2817.png) ![image](https://user-images.githubusercontent.com/3911556/236435258-56d68d50-7581-4527-bcdd-5b6e3ed53ac8.png) I have decided in favour of some restrictions. Paths must not start with a /. The only supported path separator is /. If services that require a \ are to be connected, this must be translated by the connector app. These restrictions should ensure that the file service can be exchanged without upgrading data. Assigned Workitems: Fixes #2418 Fixes [AB#559148](https://dynamicssmb2.visualstudio.com/1fcb79e7-ab07-432a-a3c6-6cf5a88ba4a5/_workitems/edit/559148) --------- Co-authored-by: Stefan Sosic <[email protected]> Co-authored-by: Jesper Schulz-Wedde <[email protected]> Co-authored-by: Darrick <[email protected]>

tinestaric

Great work on these apps! I hope they get merged in soon, it's exactly what all partners have been implementing in so many different ways!

Apps/W1/File - SharePoint Connector/app/src/SharePointConnectorImpl.Codeunit.al

Apps/W1/File - Azure File Service Connector/app/src/FileShareConnectorImpl.Codeunit.al

Apps/W1/File - Azure Blob Service Connector/app/src/BlobStorageConnectorImpl.Codeunit.al

Apps/W1/File - SharePoint Connector/app/src/SharePointConnectorImpl.Codeunit.al

Apps/W1/File - SharePoint Connector/app/src/SharePointAccount.Page.al

Apps/W1/File - Azure File Service Connector/app/src/FileShareConnectorImpl.Codeunit.al

Apps/W1/File - Azure Blob Service Connector/app/src/BlobStorageConnectorImpl.Codeunit.al

IceOnly · 2025-01-07T14:27:43Z

@JesperSchulz I still need three id ranges and app logos here. Is anything else necessary for a review to start?

Apps/W1/File - SharePoint Connector/app/src/SharePointAccount.Table.al

Apps/W1/File - SharePoint Connector/app/src/SharePointConnectorImpl.Codeunit.al

JesperSchulz · 2025-01-08T09:02:55Z

@IceOnly, let's get this on the road! Please re-ID to IDs from 4560 and up (as needed). I don't have any app icon (for now, anyway). Let's push forward without those.

JesperSchulz

Did a first round of review, mostly looking at the app setups. Let's fix these things first, and then I'll take a 2nd round, reviewing the actual code in the apps :-)

Modules/System/File Access/app.json

Apps/W1/File - SharePoint Connector/app/src/SharePointAccount.Table.al

Apps/W1/File - Azure Blob Service Connector/app/app.json

JesperSchulz · 2025-01-08T09:17:12Z

Apps/W1/File - Azure Blob Service Connector/app/app.json

+    "id":  "c9ce86fe-cb70-4b79-be03-d21856b1a4ca",
+    "name":  "File - Azure Blob Service Connector",
+    "publisher":  "Microsoft",
+    "brief":  "Enable all users to use a Azure Blob Storage to save files from Business Central.",


It's not really what this does. It adds the capability to do file operations on the Azure Blob Storage through the External File Storage module. This brief (and the other two) need a bit more work.

Apps/W1/File - Azure Blob Service Connector/app/app.json

JesperSchulz · 2025-01-08T09:21:41Z

... - Azure Blob Service Connector/app/permissions/FileStorageAdminBlobStor.PermissionSetExt.al

+
+namespace System.ExternalFileStorage;
+
+permissionsetextension 80100 "File Storage - Admin - Blob Stor." extends "File Storage - Admin"


Darn, we should have named the permission set External File Storage. File Storage suggests that you through this permission set can control the access to all files.

This will have to get fixed in BCApps, if we want to change this. Or we just close both eyes and live with it 😉

…onnectors; adjust enum and page identifiers for consistency across SharePoint and File Share connectors.

…sistency

…rePoint connectors

…ctor

tinestaric · 2025-01-09T08:17:43Z

.../W1/External File Storage - SharePoint Connector/app/src/SharePointConnectorImpl.Codeunit.al

+    var
+        ErrorOccuredErr: Label 'An error occured.\%1', Comment = '%1 - Error message from sharepoint';
+    begin
+        Error(ErrorOccuredErr, SharePointClient.GetDiagnostics().GetErrorMessage());


Thought; could we add ResponseReasonPhrase to the message if ErrorMessage is empty?

Two cases where I came across this were when I misconfigured the Sharepoint URLs, and I was getting "An error occurred." with no details, but when debugging I saw it returned a 403.

Second one was when I tried to create a file in a subfolder that doesn't exist. Error Message is empty, but 404 Not Found was still in the diagnostics.

On the one hand, I know that's not really informative for the user, but otherwise I always have to debug, to understand what went wrong...

Could also be an improvement we add later in a follow-up PR. Doesn't strike me as critical to get this PR on the road. But it's a good idea for sure. We should probably use ErrorInfo to shield the normal user from such details, while providing these important pointers in a different dimension of the error message.

JesperSchulz

Coming along nicely! I couldn't find anything serious, but would suggest to revisit the permissions.

JesperSchulz · 2025-01-09T09:43:41Z

...File Storage - Azure Blob Service Connector/app/permissions/BlobStorObjects.PermissionSet.al

+
+    Permissions =
+        table "Blob Storage Account" = X,
+        codeunit "Blob Storage Connector Impl." = X,


Why not use inherent permissions/entitlements on most of these objects. Not ever would a system admin be interested in controlling the execute permissions of e.g. an Impl. codeunit. Let's only have the permissions configurable, which an admin actually could care about!

Apps/W1/External File Storage - Azure Blob Service Connector/app/src/BlobStorageAccount.Page.al

Apps/W1/External File Storage - Azure File Service Connector/app/src/FileShareAccount.Table.al

Apps/W1/External File Storage - Azure File Service Connector/app/src/FileShareAuthType.Enum.al

Apps/W1/External File Storage - SharePoint Connector/app/src/SharePointAccount.Table.al

JesperSchulz · 2025-01-09T09:58:02Z

.../W1/External File Storage - SharePoint Connector/app/src/SharePointConnectorImpl.Codeunit.al

+    var
+        ErrorOccuredErr: Label 'An error occured.\%1', Comment = '%1 - Error message from sharepoint';
+    begin
+        Error(ErrorOccuredErr, SharePointClient.GetDiagnostics().GetErrorMessage());


Could also be an improvement we add later in a follow-up PR. Doesn't strike me as critical to get this PR on the road. But it's a good idea for sure. We should probably use ErrorInfo to shield the normal user from such details, while providing these important pointers in a different dimension of the error message.

…int Account tables; fix enum value formatting in Ext. File Share Auth. Type

…ePoint connectors; include inherent permissions and entitlements

…ePoint connectors to include "Ext." prefix

IceOnly added 5 commits May 3, 2023 15:17

Initial commit

7584061

Add File Browser

6ecfacc

Add File System Codeunit

25bc19c

Add documentation and small changes

959f05e

Fix up action

4bb4080