You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Next »

Status

WIP

Stakeholders
Outcome
Due Date
Owner

SAUSSOL, Olivier 

Solution/Domain/Data Architect

TRIFFAUX, Eric PASQUALI, Laurent alexandre.mathe@syensqo.com

Most of the lab sites are equipped with File Server atop iOMega NAS, which is in some cases if malfunctioning also are running at the limit for adding more hard disks to accommodate new demands for storage and been compliant with data storage regulations.

[ TL;DR]... tbd

Use cases

  1. Lyon/RICL
    1. OpenLab instruments generate files at hundreds GB scale for some techniques which demands local computer power to load in stand alone applications for expert analysis, this magnitude size also makes it impossible to load these files to cloud repository as well as creating difficulties in being compliant with regulatory terms for storage.
  2. Shanghai
    1. Waters instruments demands 12TB space to store application data. This storage magnitute reaches the local limitation for LabPC disk and also for local network in the iOMega NAS.
  3. Bristol
    1. Waters instruments demands 12TB space to store application data. This storage magnitute reaches the local limitation for LabPC disk and also for local network in the iOMega NAS.


Questions and Concerns


Architectural Significant Requirements

  • Data rotation
  • Retention
  • Export Control/Cyber Sec
  • Access Control
  • Data transfer latency SLA
  • Data Consumers (data structure/data model)

Impediments and Blockers


Tradeoff analysis

  • Alternatives
    • XYZ
    • YZX 
      • Sensitivity Points
      • Risks
      • Non Risks
      • Architectural Approaches


Quick-Wins

The criticality of the problem demands that there be at least a temporary solution as an alternative to mitigate the immediate impact of losing sensitive data for the business due to the local storage limit at the current date.

Some questions that can facilitate the analysis:

  1. What type of search criteria is used on historical data?
  2. What retrieval is done on this found data?
  3. What type of processing is done on the found and retrieved data?
  4. Is it possible to parse this data?

Possible alternatives to consider in advance:

  1. Promote historical data close to the (AWS Landing Zone) ACD Labs domain so that it can be ingested for later analysis in reports
  2. Promote historical data close to the (GCP) Lab-Booster domain so that it can be ingested for later analysis in reports
  3. Promote historical data to the Azure Fabric Lakehouse so that it can be ingested for later analysis in reports


Design Solution Proposal


Meeting Notes

[SoW] LabPC - Storage for lab-local data at scale (Olivier SAUSSOL, Mijajlovic, Julie, Tiago Oliveira)

  1. LabPC Storage Study
    1. SoW
      1. Assess the issue
        1. Scenarios (business needs?)
          1. *Pictures* for real case situation (disks at the floor...)
          2. Worst case scenarios
          3. Application data, storage for long term, later data combination cross apps
          4. Instruments categories (data volume, SLA,...)
          5. (inventory for storage) LABPC Storage needs consolidation https://docs.google.com/spreadsheets/d/1U-d6W2LEGV9mz4XK9hBmHg1FTZY_-oYxlkXKkukgSNQ/edit?gid=0#gid=0
          6. Total storage needed for now and the forecast?
        2. Impact
          1. Reaching limit of storage
          2. User storing data on inappropriate devices (NAS, - shadow it)
        3. Risks
          1. 1.Data loss
          2. 2.Data steal
          3. 3.Shadow IT
          4. 4.Export data control/legal regulations concerns
          5. 5.Business continuity
        4. Cost
          1. Getting expensive to maintain/extend disks
        5. Impediments
          1. Hard to standardize instruments
          2. Blocked to onboard new instrument
        6. User complains
          1. Bolate: 77 pcs, storage not appropriate
          2. Shanghai:
          3. "Mark´s use case: large file generated
      2. Evaluate Alternatives
        1. Meeting Stakeholders
        2. Business strategy
        3. Skills demanded for the assessment
        4. Solution Vendor providers
        5. Short Term solution (quick-wins)
          1. Already some in place - *highlight quick-wins ongoing
            1. Distributed storage on LabPC (disks): store on LabPC by convenience to avoid losing data
            2. Leveraging local File Server (*Shanghai - non standard storage)
            3. Cloud storage for historical LabPC data - it depends on application readiness for Virtualization deployment
              1. Recall AWS gateway for data - PoC (ask Khemaies)
              2. PoC for instrument application virtualization (pick that one with larger data volumetry; ask Olivier)
              3. Consider ACD Lab (as Mark Kwasnik) as the interface to run analytics - in case instrument application not ready for virtualization
        6. Long Term solution
      3. Outcome
        1. Presentation
          1. (1st phase)structure scenario (deadline ) [Julie, Olivier, Tiago]
            1. touch points weekly basis meeting (30min) March (Wednesday 10:30)
          2. (2nd phase)engage expert per domain for the solution alternatives
            1. Decision to be taken
            2. Business Case
            3. Project organization
            4. Solution Architecture Proposal

References

LABPC Storage needs consolidation https://docs.google.com/spreadsheets/d/1U-d6W2LEGV9mz4XK9hBmHg1FTZY_-oYxlkXKkukgSNQ/edit?pli=1&gid=0#gid=0

Presentation https://docs.google.com/presentation/d/1UE4-b8e0GxHoxlVHJomt54JpcWwCv5Uh48Tw3qLA41o/edit#slide=id.g33d7e03e36c_0_0



  • No labels