Look sharp

Some usefull Linux commands to keep the site updated

Deploy .NET project

rsync -avz -e 'ssh' /mnt/c/Users/<project>/<user>:/var/www/<project>

Update system

sudo apt update && sudo apt full-upgrade

Split text on right most space before limit is reached
Use to split text on several lines in receiving document, eg edifact LIN-IMD-C273 has several 7008 text element

//    -----------------------------
//    Peter Lykkegaard, 22 mar 2019
//    -----------------------------
//    Split long note on right most space before limit is reached
//    TODO! Check parameters if they are of valid type/content
//
//    Name: splitNoteBySpace
//    Description: Method to split a given string by latest space before length limit
//    Parameters
//        input$, alphanumerical / String which needs to be split
//        max$, alphanumerical / Max characters on each line
//    Output, alphanumeric / array / Array of strings to return
 
local lvOutput$[];

#importJavaStart
    import java.util.regex.Matcher;
    import java.util.regex.Pattern;
#importJavaEnd
 
#javastart

    String input = _StrVar_PINPUT.getString();
    int max = (int)_NumVar_PMAX.getNum();
    int len = input.length();
    
    String regex = "(.{1," + max + "})(?:\\s|$)";
 
    Pattern pattern = Pattern.compile(regex, Pattern.MULTILINE);
    Matcher matcher = pattern.matcher(input);
    
    _StrVar_LVOUTPUT.getJuVar(0, 2).setString(input);

    if (len > max)
    {
        int idx = 0;     // Entries in output   
        while (matcher.find()) {
            for (int i = 1; i <= matcher.groupCount(); i++) {
                _StrVar_LVOUTPUT.getJuVar(idx, 2).setString(matcher.group(i).toString());
            }
            idx++;
        }
    }
#javaEnd
     
exitProc(lvOutput$);

Validate EDI (XML, Edifact and Ansi X12)  

Site

• EDI and Content File Validator - validatefile.dk

Functionality

• Multiupload
  • No data is stored on the server
  • MIME type is detected based on file content
    • It is possible to upload without file extensiom
    • All other files than text (edifact / ansi x12) or xml are skipped / not supported
• Validate structure (eg schema validation)
• Validate business logic (eg schematron validation)
• XSLT Styling, readable document in HTML
• Calculate content and compare with values in eg an invoice

Milestones

• Multiupload based on Dropzone JS: https://www.dropzone.dev/
• Implemented Mime-Detective in class factory: https://github.com/MediatedCommunications/Mime-Detective
• Prepared and created concept to post files several times from Dropzone JS queue
  1. Upload to local client (browser) 
  2. Send file content to server to either
     1. Check and validate content
     2. Check business logic
     3. Style document
     4. Calculate prices and compare with document values
• Use RaspConfiguration (Converted to JSON)
  • Detect document type to prepare for XSD (logic) and Schematron (business) validation 
  • LINQ lookup to get correct DocumentTypeConfig
  • Schema validation with PSVI
    • https://www.w3.org/XML/2002/05/psvi-use-cases
• Cookie consent
• AntiforgeyTokens
  • Antiforgery: https://learn.microsoft.com/en-us/aspnet/web-api/overview/security/preventing-cross-site-request-forgery-csrf-attacks
  • JSON: https://stackoverflow.com/questions/32287901/how-do-i-use-antiforgerytoken-with-dropzone-js-and-mvc-5-with-vanilla-js
• NuGet package: MimeDetective (OK)
• Mime Mapping (Done): https://www.garykessler.net/library/file_sigs.html
• Alternative API content check: https://www.appsloveworld.com/csharp/100/139/alternative-to-findmimefromdata-method-in-urlmon-dll-one-which-has-more-mime-type (Skipped)
• When XML Reader is opened reset the stream to the starting point: stream.Seek(0, SeekOrigin.Begin);
• OIOXML schemas, validation and styling on Gitlab: https://rep.erst.dk/git/openebusiness/library/java/-/tree/6843728b8e72108eeb1dc2a30ed324fb61cb4c2b/dk.gov.oiosi.examples.client/resources
• Display XML content on web page
• Verification test on Kestrel and local IIS

Current task

• Scoped the project, only a few XML and EDIFACT versions will be implemented
  • OIOUBL (danish adaption of the UBL standard)
  • Peppol BIS
  • Standard eg EDIFACT D96 are considere in a limited version as a start as a proof of concept
  • cXML and xCBL are a possibility
• Store validation classes / objects in memory cache to use on subsequent requests
• XSLT transformations uses SaxonHE
  • All stylesheet versions are supported
  • Included stylesheets or XML documents are supported
• Documentation of source code (XML documentation)
  • SandCastle is used to generate HTML or MD documents
  • Add content documentation to source code

Release

•  WIP
  • Jun 23: Only local test / development atm, date for a first beta release not estimated
  • Oct 23: Slow process with local test / development, getting closer to a beta release 
  • Dec 23: Testing, prepare for release on VPS with Ubuntu
  • Jan 14: Project running on VirtualBox / Ubuntu 23.10 / NET 8
    • NGINX as Reverse Proxy in fron of Kestrel web server
  • Jan 14: Test memory usage to validate 8 Gb mem limit
  • Jan 23: Released on https://www.validatefile.dk
    • VPS with 2 CPU, 8 Gb mem running Ubuntu 23.10 server and .NET 8
  • Feb 07: Small adjustment, preparation for xCBL and cXML validation, About page added

To do list for project: 

• Create Edifact Icon in SVG
  • Tools needed:
    • Inkscape SVG editor
    • Fontforge: Create WOF2 package
• Validate file content
  • javascript:
    • https://www.dotnetexpertguide.com/2011/05/validate-uploaded-image-content-in.html
    • https://gist.github.com/aziz-blr/fdef65aed6d60729b47f1f52bd6bd0af
  • XML schema validation: https://learn.microsoft.com/en-us/dotnet/api/system.xml.schema.extensions.getschemainfo?view=net-7.0
    • Will be replaced with SaxonHE libraries
• Additional upload for schematron valditation, styling and calculation:
  • https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/Ajax-JavaScript-file-upload-example
• Document examples from Pagero:  
  • https://rep.erst.dk/git/openebusiness/common/-/tree/master/resources/Documents?ref_type=heads
  • https://pagero.github.io/business-documents/invoice-creditnote/
  • https://github.com/pagero/integration-resources/tree/main/example-files
• Document styling
  • Idea for future output from html stylresheets
    https://github.com/unimaze/unimaze-peppol-stylesheets?tab=readme-ov-file
• Display XML content on web page: (Done)
  • https://learn.microsoft.com/en-us/previous-versions/aspnet/13ftcwy9(v=vs.100)
  • https://blorgh.wordpress.com/2006/10/17/displaying-xml-in-the-net-webbrowser-control/
  • https://learn.microsoft.com/en-us/answers/questions/1181957/show-xml-in-webbrowser-control
  • Use ContentResult in Controller, open new window in JS and paste html content (POC done)
• Edifact example source / projects
  • https://github.com/8/Edifact
  • https://github.com/indice-co/EDI.Net
  • https://www.nuget.org/packages/indice.Edi (same as above)
  • Extras I need to check later: https://michaelachrisco.github.io/Electronic-Interchange-Github-Resources
  • Former EDIFabric: https://github.com/EdiWeave/EdiWeave
    • Characterset / Encoding:
      https://blog.sandro-pereira.com/2009/08/15/edifact-encoding-edi-character-set-support/
• favicon (Done)
  • https://stackoverflow.com/questions/4014823/does-a-favicon-have-to-be-32%C3%9732-or-16%C3%9716
• Popup window with validation messages (Abandoned)
  • https://stackoverflow.com/questions/1216114/how-can-i-make-a-div-stick-to-the-top-of-the-screen-once-its-been-scrolled-to
• Milliseconds elapsed (Done)
  • https://stackoverflow.com/questions/41632942/how-to-measure-time-elapsed-on-javascript

Sandcastle

• https://www.red-gate.com/simple-talk/development/dotnet-development/taming-sandcastle-a-net-programmers-guide-to-documenting-your-code/

Use local postman for testing?

• https://weblog.west-wind.com/posts/2017/Sep/14/Accepting-Raw-Request-Body-Content-in-ASPNET-Core-API-Controllers?Page=2
  Working, use vaiable name from controller as property name in postman

Unit test

• https://www.toptal.com/qa/how-to-write-testable-code-and-why-it-matters

Check for vulnerability

• https://pvs-studio.com/en/
• https://itnext.io/vulnerabilities-due-to-xml-files-processing-xxe-in-c-applications-in-theory-and-in-practice-f94912216093

GDPR (mostly done)

• Cookie consent: https://zetbit.tech/categories/asp-dot-net-core/10/creating-a-cookie-policy-consent-popup-in-asp-net-6
• (GDPR) support in ASP.NET Core: https://learn.microsoft.com/en-us/aspnet/core/security/gdpr?view=aspnetcore-7.0

Information or questions: Please contact plykkegaard at gmail dot com

Edit IDOC document properties

Once the IDOC definition is imported in the Mapping Designer make sure to set Max Occurences on the document root (DC1)
Use unlimited (-1) or 9999

• IDOCs are send from SAP to a queue with append
• File is read and mapped to edifact
  • Header is set in New Mapping and footer in Mapping End

An utillity to get a random UUID
This is a procedure to use in Seeburger BIC Mapping Designer

/*  -----------------------------
    Peter Lykkegaard, 25 apr 2023
    -----------------------------
    procedure getUUID
  
    Parameters:
        None
    Output
        String
  
	Create random UUID
    ----------------------------- */
 
#importJavaStart
    import java.util.UUID;
#importJavaEnd

local output%;

#javastart
    _StrVar_OUTPUT.setString(UUID.randomUUID().toString());
#javaend

exitProc(output%);