Objective
This document describes in detail the Monitoring activities to be performed by the agent in different shifts, days of the week, D4 period etc
Introduction
Different working shifts
The following are the working shifts for BW agents :
S1 - Working hours between 7AM and 4PM IST
S2 - Working hours between 1PM and 10PM IST
S3 - Working hours between 10PM and 7AM IST
Tool used for process chain monitoring
We use “ST13” T-code to monitor the status of all chains running/failed in BW ( No need to use Firefighter to use ST13 transaction code (all BW developers have ST13 by default in the BW developer role) , the process chain status has to be checked frequently to identify any failures or long running jobs.
Tcode - ST13:
Tool name - bw-tools
Feature list - Process chain analysis
The following variant can be used to get the list of all running and erroneous process chains for the current day.
Variant name: RUNNING_FAILED (All running and failed chains)
Note: All the process chains running in the system have to be monitored and corrected in case of any errors.
D4 period (Month- End Closure)
Closing period: (D-1 to D+4)
i.e D-1, D1, D2, D3, D4 where D-1 is the last working day of a Previous month and D1, D2, D3,D4 are the 1st,2nd,3rd and 4th working day of a Current month respectively.
Working Days : MON, TUE, WED,THU, FRI
Non -Working Days : SAT , SUN and any Bank holiday in France.
The D4 period is critical as several users use BW reports to perform the month end closing activity during this period and we have to take extra care to make sure all the process chains run smoothly during this period.
Fixed Asset Supplier (FAS) and Lease Interface
Agents must Manually check if the files have been sent to Source system or not during the closing period.
It is critical that we make sure we are generating and sending the files
WP1 = /exploit/BW/RTR/FAS/FAS_data_from_WBP_to_RCS.csv,
PF1 = /exploit_ERP/BW/FAS/FAS_data_from_WBP_to_SOLVAY.csv
PI1 = /exploit/BW/RTR/FAS/FAS_data_from_WBP_to_CICC.csv
A mail has to be sent to the following reciepients once the D4 chain has run and the files have transferred, the screenshot of the updated file in AL11 in the respective systems has to be included in the mail:
· At D-1, one single update related to the 3 runs at the end of the day (as soon as last run has properly ended up)
· At D1, one single update related to the 6 first runs (as soon as 6th run has properly ended up)
· At D2,
. One first update including last run of D1 + first run of D2 (as soon as first D2 run has properly ended up)
. A second update including D2 second and third runs (as soon as third D2 run has properly ended up)
. Not necessary to get an update for D2 runs #4, 5, 6 and 7.
"LANGUE, MATHIEU" <mathieu.langue@cgi.com>,
"Rayala, Brahmam" <brahmam.rayala@cgi.com>,
"Sammeta, Natraj Aditya Kumar" <natrajadityakumar.sammeta@cgi.com>,
"JAGUELLO, Dimitri" <dimitri.jaguello@cgi.com>,
"Vathadi, Chandra Sekhar V" <chandrasekhar.vathadi@cgi.com>,
"chandrasekhar.vathadi-ext@solvay.com" <chandrasekhar.vathadi-ext@solvay.com>,
Radha Eepuri-ext <radha.eepuri-ext@solvay.com>,
"Yeturu-ext, Amareshwar" <amareshwar.yeturu-ext@solvay.com>,
Harsha Uppasettyraghava-ext <harsha.uppasettyraghava-ext@solvay.com>
"Pulluri, Venkateswarlu" <venkateswarlu.pulluri@cgi.com>
Detailed explanation and steps to be follwed are mention in the link below.
Monthend CAPEX flat file not downloaded on WP1 or PF1 or PI1 systems
IN CASE OF AN ISSUE:
In case of any file transfer issue for 3 flows in FAS interface ie. from source(WBP) to MFT or from MFT to target(CICC/RCS/SOLVAY) OR MFT job starts failing due to connection issue , a freshdesk ticket will automatically generate and assign to IS-CAPG-B2B/IS-CAPG-MFT which is taken care by webMethods/MFT Support team (Support.XML@solvay.com)
At the same time, an alert email notification will also be sent to the below teams each time a freshdesk ticket is generated.
To: support.xml@solvay.com
Cc : sbs-is-appli-finance-reporting@solvay.com,
sbs-is-appli-techno-report-bw.cc@solvay.com,
SBS-IS-Appli-Techno-Report-BW.CGI@solvay.com
However during closing as it this is very critical BW team should c heck if the file are well transferred.
In any issue please create a new freshdesk ticket with the subject SAP WBP - Issue WebMethods Active Transfer (MFT) - <name of the file transferred>
Updating D4 sheet
Below is the folder and you have to update the sheet for the current D4 closing month:
https://drive.google.com/drive/folders/18gAFMy3OCP4Att8myTc5A1EclUh2cWjB
Daily activities:
Important process chains
PC_FREQUENTLY_LOAD_MAIN
Control D4 by global filter with Steam = MULTIPLE
This chain runs 3 times daily in non-closing period and will run 7 times in Closing Period.
During closing period (D-1 to D+4): Rule = M_CLOSE
During non closing period: Rule = NOM_CLOSE
The chain usually runs around 60 to 90 minutes, it is important to make sure that the run is completed within 120 minutes, especially during the closing period as the process chain will not start if the previous run is still in progress.
parameter to setup time and type of loading (Complete or Minimum)
Rule = WORKDAY --> set working day of month end up to working day 4th
RSP_DAILY
RSP_DAILY is one of the important process chains that contains several local chains related to master data, COLMAR, SPRINT, Global sales etc.
RSP_DAILY runs only on weekdays and starts at 12AM CET (midnight).
PC_GL_SALES_RAISE_EVT003 will wait for the events to be triggered
- Z_EVT_END_GL001 ==> will be triggered from PC_SD_MAIN=>PC_GL_SALES_RAISE_EVT001
- Z_EVT_END_GL002 = => Will be triggered from PC_GLOBAL_SALES_MAIN→PC_GL_SALES_RAISE_EVT002
PC_GL_SALES_RAISE_EVT006 will wait for the following events to be triggered
- Z_EVT_END_GL005 => Will be triggered from PC_GLOBAL_SALES_MAIN=>PC_GL_SALES_RAISE_EVT005
- Z_EVT_END_GL004 ==> will be triggered from PC_SD_MAIN=>PC_GL_SALES_RAISE_EVT004
PC_COLMAR_05
PC_COLMAR_05 is one of the important chains to update the COLMAR portal and this chain starts when the following chains completes:
- ZPC_MD_ATTR_RCS
- RSV_COOM_CCA_OTHER (scheduled start at around 2AM CET)
- RSP_PM_SOLVAY (scheduled start at around 2AM CET)
- RSP_COOM (starts after completion of ZPC_MD_ATTR_RCS)
- RSP_PM (starts after completion of ZPC_MD_ATTR_RCS)
The following variant can be used in “ST13” to check the status of dependency chains of PC_COLMAR_05
Variant name : COLMAR DEP (Colmar chain and its dependent)
COLMAR daily status mail
- Start of COLMAR chain mail - Once the process chain PC_COLMAR_05 starts, a mail has to be sent, referring previous mails for subject, DL and body of the status mail.
- End of COLMAR chain mail - Once the process chain PC_COLMAR_05 completes, a mail has to be sent, referring previous mails for subject, DL and body of the status mail.
- Ad hoc mails for COLMAR
- COLMAR not started
If the chain -” PC_COLAMR_05” has not started by 5:30AM CET, We have to send a mail to COLMAR team by saying that Colmar chain is not started due to xxxxxx issue, the DL should be the same as you would send to Colmar start/end mails. - COLMAR is in error
If the COLMAR chain is in error and it is not rectified we have to send a mail to the COLMAR team right away mentioning that the chain is in error and necessary actions are being taken, the DL should be the same as you would send to Colmar start/end mails. - COLMAR error corrected
Once the error is corrected, COLMAR team has to be notified and the ETA of the chain completion has to be provided, the DL should be the same as you would send to Colmar start/end emails.
- PM-MECANO - Recovery Procedure for Issues on BW Data Loadings
The Colmar team has to be updated with the current status of the loads every 1 hours in case of any failures encountered, once the chain completes you would send the COLMAR ended mail that you would send daily.
Important : COLMAR start and end mails have to be sent everyday while adhoc mails are to be sent only during issues.
RSP_DAILY_02
This chain is triggered by an event and this event is automatically triggered after completion of the below jobs in WP1 system
JOBS , TCODE SM37
Job in WP1 to trigger PC
PROGRAM :: ZWPP_TRIGGER_EVENT variant WBP
JOB Name in WP1: PP_TRIGGER_EVEN
Before run the BW trigger program these programs need to complete first. In case of loading error in Bw, need to check on these jobs log
PP_AP_ZWPP31B_DAY PP_AP_ZWPP31B_WEEK = program ZWPPR01800B variant DAY_AP / WEEK_AP users OPSCHEDULE
PP_EU_ZWPP31B_DAY PP_EU_ZWPP31B_WEEK
PP_NA_ZWPP31B_DAY PP_NA_ZWPP31B_WEEK
PP_SA_ZWPP31B_DAY PP_SA_ZWPP31B_WEEK
After completion of the above jobs in the WP1 system, the chain “RSP_DAILY_02” will trigger in the WBP system.
Need to check previous logs to find out the expected time of start for these jobs.
In case if the chain does not start in WBP even after the completion of jobs in WP1, the chain has to be manually started in WBP (the chain has to be triggered manually only before 4PM CET, no manual triggers after 4PM CET)
In case if there are any failures in the jobs, a ticket has to be raised and assigned to IT Indus Stellar L2 ( Partner). Monitor team should first check on the WP1 log if the job fail because of what
For example of new plants. It will display what plant cause the issue
Solution:
- CGI has to send ticket to I&P team (IT Indus Stellar L2 ( Partner)) to check why it is error. (contact the data admin of the GBU to correct the inconsistency eg. sylvie.lebrunet@solvay.com Data Admin for Novecare)
- Edit variant for the job to exclude the plant to allow the PC run normally
- Once I&P team done, please include the plant to the job again.
Dynasys chains
The detailed instruction for monitoring Dynasys chains, sending ad hoc mails are described in the below wiki link, appropriate actions are to be taken as per the below document:
https://wiki.solvay.com/pages/viewpage.action?pageId=328106012
All the failures related to dynasys loads are to be recorded in the below file:
Important: the Chain PC_DPS_DYNASYS_03 runs every saturday, incase of failure it should not be repaired on Monday, wejust need to inform Dynasys team on the failure.
MECANO Chains
Following are the important chains related to MECANO loadings:
1.PC_PM_MECANO_01 is the main load which will run from Sunday to Thursday @08:30PM CET (the sunday chain status has to be checked by the S1 agent on monday)
2.PC_PM_MECANO_02 is the weekly load which will run only on Monday @ 01 AM CET
3. PC_PM_MECANO_03:Daily Runs at 02:00 AM CET expect on weekends
MECANO loads are snapshot loads, detailed procedure on recovery in the below document:
https://drive.google.com/file/d/18qbfYpKMngg3aCjd0O-BSX0GWbT2M3wNqkVfvbAl4HI/view
SPRINT and GLOBAL SALES Process Chains
PC_SPRINT_DAILY is the main process chain related to SPRINT loading and it is a local chain in RSP_DAILY.
We have chains related to GLOBAL SALES in RSP_DAILY as well as chains running outside of RSP_DAILY.
PC_GLOBAL_SALES_MAIN ( Runs daily after the event Z_EVT_END_TIERS_RHODIA) , PC_GL_SALES_MAIN_PO etc are some of the global sales chains.
Dynasys Interface chains are dependent on PC_GLOBAL_SALES_MAIN.
If we have any delay in completion of this chain or not started for any issue we have to inform Diogo Paiva on global sales loads
Performance report (PUPE chains)
We need to follow the below steps for any load related issues on "PUPE" (Performance report) ,
Data will load through a flat file (AL11).
Path : /exploit/BW/BOZAR/ConvergenceToBW.csv
Process chains:
PC_PUPE_04
PC_PUPE_03
PC_PUPE_02
Launched by event Z_EVT_PC_PUPE (triggered by webmethods application with function module Z_TRIGGER_EVENT)
1) If any load failed due to invalid characters issue, then we have to contact sales force team (sylvie.severini@solvay.com/ yiting.kuo@solvay.com / abdou.daayf@solvay.com ) or GPS.PUR@ solvay.com in order to correct the format/data.
If no one is available, then we have to contact GPS.PUR@solvay.com team.
2) If the file is corrected on the same day by the sales force team, Then we need to perform manual load in BW.
Please find the below procedure for manual load
https://wiki.solvay.com/pages/viewpage.action?pageId=20882843#BWPTP-SPRINT(includingSRM7,PUNCH,PurchasingVBF)(WBP)-Loaddatamanuallyincaseofissue
3) After loading the data to BW, We have to inform the QV team to load the data. Data will be appear good in the dashboard after loading by the QV team.
4) Load data manually from SalesForce
(a) Connect to SalesForce. If you do not have access, contact Sylvie Severini to have an account created. If she is unavailable, send a mail to gps.pur@solvay.com
(b) In Dashboards, open Report "CONVERGENCE><SPRINT".
(c) Run Report, wait for the page to reload, then Export Details.
More info on other issues : BW PTP - SPRINT (including SRM7, PUNCH, Purchasing VBF) (WBP) /!\ Obsolete /!\
RPC_FC2_BUDGET
This is a Weekly Chain, runs Every week on Wednesday at 7:00 CET
It will take around 8 to 9 hours to finish the loads in the chain
Chemplex Monitoring
PC_GBR_14 - Runs Daily + SAT & SUN also - Z_EVT_PC_GBR_14 in BW will trigger the PC.
Normally this chain will start at early morning around 1.59 AM CET and will complete in 15 to 20 mins
PC_GBR_15 - Runs Daily + SAT & SUN also - Z_EVT_PC_GBR_15 in BW will trigger the PC
Normally this chain will start at early morning in between 1.30 to 2.00 AM CET and takes around 20 mins to complete the load
SOLSTICE chains in WBP
There are several chains related to SOLSTICE which are mainly run to feed open hub destinations.
Some example SOLSTICE chains:
Common errors in SOLSTICE:
- When ever we face RFC issue with SOLSTICE loads
Take the Open hub name(OHxxxxxx)
Go to SM59 transaction
CHeck for LSINFSBW in TCP IP and check connection parameters
"LSINFSBW" :
If there is a RFC/connection issue, we have to contact the SOLSTICE team.
(Animesh shahi, Craig Wannamaker, Sunil shah). --> For PC failures, Animesh is the main person to contact.
2. DTP will fails due to lock issue :
For this, We have to delete the old request and need to trigger the DTP again.
As this is Open hub, we will not have access to delete the Red request,you have to display DTP variant, and click on execute manually, hence it will pop up like if you want to proceed, you have to delete the old request, We will click on “Yes”(Proceed) so that old request will be deleted followed by a pop-up to execute the DTP, by clicking on “Yes”, DTP will run.
CPX Chains
CPX is a mini BW system of SOLSTICE, we have to log in to CPX system (Client 400) to monitor the chains.
You may use the T- code RSPCM to monitor the chains, in case you lack access to ST13 T-code, the following chains are already available in RSPCM.
Chains to be monitored
RSP_DAILY_GLOBAL = 19:00:00 EST ( only workday )
RSP_DAILY = MidNight EST ( Only Workday )
PC_CO_PC_MM_04 = Monthly . This is event driven. (Usually runs in the first week of the month)
Actions to be taken when process chain is in error in CPX
We will ONLY MONITOR , In case of Error WE ONLY COMMUNICATE But we DO NOT RECTIFY it .
Communication shall be done through mail n below format
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
To : Craig Wanamaker <craig.wanamaker@solvay.com> , Sunil Shah <sunil.shah-contractor@solvay.com>
CC : Venkateswarlu Pulluri-ext <venkateswarlu.pulluri-ext@solvay.com> , Sunil Kumar <sunilkumar.n@cgi.com> and all team members of the support team.
Subject : CPX - PC Error - << DD-MM-YYYY>>
Main Chain : << Technical Name >>
Sub chain : << Technical Name >>
Variant : << Description >> - << Technical Name>>
Error : Details of Error ( If possible provide screenshot )
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Daily status mails
Process chain status mails
Please refer to sample mails sent previously for subject, DL and body of the status mail, incase of any ongoing issues, the status of the issue and actions taken has to be mentioned in the body of the mail.
Two process chain status mails have to be sent.
1st Update - 09:00 AM CET
2nd Update - 13:00 PM CET
Please Refer to How To Monitor PC and Update Status File for detailed explanation and Step by Step Process.
Broadcast
We need to update the Broadcasts status in the PC monitoring sheet daily before sending first status mail.
Please find the below steps on how to check the broadcast status in BW
Step 1 : Goto Tcode RSRD_ADMIN
Step 2 : Click on Log display
Step 3 : Provide date and User field values date should be yesterday and today date
step 4: Click on execute
step 5: If any of the broadcast status is red we have to take actions based on the error
You can find the more details of the Broadcast in SOST tcode
Incase of any issue in broadcasts, appropriate action has to be taken.
Broadcast for currency conversion check
The following broadcast is scheduled to check if there are any issues in currency conversions, this is sent to the support team.
If any * value in measures ,do analysis and inform "Duarte Costa" accordingly .
The purpose of this broadcast is to identify if there is any issue with exchange rates and try to rectify it before users raise the issue .
Jobs running in WP1
The job - LIS-BW-VB_APPLICATION_17_400 runs in WP1 and it has to be monitored and it has to be updated in the status mail as well.
Monitor Background jobs in SM37 screen
ST22 Dumps
Everyday ST22 dumps are to be checked for WBP system for today and yesterday, this has to be done before sending the status mails and if there is anything alarming, need to check further and inform IBM if required
Service monitoring tickets and Repetitive Errors
Service monitoring tickets
Service monitoring tickets in Fresh Desk are created automatically by the system when there is a Load/ Variant in the process chain step failed - Not for every load - it is configured to only critical loads/variants.
These tickets would have the subtype as “Service monitoring” and would be assigned to the IS-CGI-L2-BW Group without any agent. Domain Lead will be assigned to L1-BW Member. \
It is the responsibility of the L1 Agent to close the ticket on the same day.
Agents need to update the following in the ticket before closing it.
Process chain : << Technical Name of PC >>
Variant Type :
Variant : << Technical Name >>
Error Details :
Action taken :
Repetitive errors
If there is a same/similar issue that is happening frequently, the agent has to work on the permanent fix so that the failures don't happen in the future, incase of difficulties of finding RCA or permanent fix, it has to be brought to the team’s notice.
IMPORTANT: It is the responsibility of the L1 agent to make the permanent fix, in case if it is sent to someone else in the team, the agent has to continuously follow up and make sure that the permanent fix is applied.
Incident logging
All the process chain failures have to be logged in the below file, there could be cases where there is a failure and there are no service monitoring tickets created, irrespective of the creation of service monitoring tickets, all the process chain failures have to be logged.
Incident log file:
https://drive.google.com/file/d/1wktL_kEb9H5xyhCu9ruiooQfm708Qg3CpU2c5fux93M/view
You can find the detailed instruction for incident logging in the below file:
https://docs.google.com/spreadsheets/d/1Ev8vGONfjEXp7WdIp6_cC6B-29qfHOQhIMRAoUifx2Y/edit#gid=4
Shift Handovers
S1 to S2
Once the agent logs in the S1 shift, the agent has to check if there are any pending process chain issues from the previous date and on mondays and also follow up if there are any pending issues reported by S3 agent.
All the pending issues have to be reported to the S2 agent before the S1 agent logs off.
S2 to S3
The S2 agent has to work on any pending issues reported by the S1 agent and the S2 agent has to report any pending issues to the S3 agent before logging off.
S3 to S1
The S3 agent has to work on any pending issues reported by the S2 agent and the S3 agent has to report any pending issues to the S1 agent before logging off.
Weekend activities:
SATURDAY - only 6 hours Support ( 06:00 AM CET TO 12:00 PM CET)
Status mail has to be sent at 12:00 PM CET , one L1 member will be working on Saturdays to monitor , correct and send status mail.
SUNDAY - NO Support
The status mail for the process chains that run on Sunday has to be sent by the S1 agent on Monday morning.
IMPORTANT : “S1 agent has to check the status of all the chains that have started on Saturday and Sunday on Monday morning and make necessary actions in case of any long running or failed loads”
https://drive.google.com/file/d/1AORWL4Hi4rTdB6ca09GfLiL9kM1eK-yHLbIllZWoD8Q/view
WEEKLY Process Chains
We have chains that run on the weekly/ monthly basis, you will find the details of the same in the status mail file and it has to be checked accordingly.
These chains have to be monitored on the appropriate day/date of the week/month respectively.
Example weekly chains in the sheet:
MONTHLY Process Chains
Example monthly chains in the sheet:
Best practices
Never skip a process chain step without taking necessary approvals, it could have an impact in the data and could create inconsistencies.
Keep checking the “ST13” tool frequently to identify issues/long running or stuck jobs quickly and act accordingly. => No need to use Firefighter by using ST13 transaction code (all BW developers have ST13 by default in the BW developer role)
Always try to find the RCA and permanent fix for any failures.
For failures due to invalid characters, even though we correct the data by removing the invalid characters from PSA/error stack in BW, it is mandatory to get the invalids corrected in the source ECC as well for both delta and full loads, if you see that there are frequent failures due to invalid characters in a specific flow, function module has to be implemented to tackle this.
Always keep the concerned people posted on the update and continuously follow up corresponding teams for resolution.
Ad hoc mails have to be sent as soon as possible to keep everyone informed about the situation.
In any cases where you are unsure of the correction, please reach out to the team to avoid any issues.
Appendix
PM-MECANO - Recovery Procedure for Issues on BW Data Loadings
Contacting Basis Team ( InfoSys)
AWS Service Management - Incident, Service Request and Change Management Process v2.3
If the incident is related to the infrastructure of applications hosted at AWS, authorized users can report this incident to the Infosys service desk:
Write an email to the Infosys Service Desk: infycisss@service-now.com , solvaycloudops@infosys.com with information about applicati on(s) impacted, i.e. server, application name and clear description of the issue incl. error message, attach useful screenshots if applicable
InfoSys Team handles team handles issues related to SAP servers which are moved to AWS Cloud Environment. Like WBP, PF1, PI1 and its Dev, QUA and PRD systems.etc systems, we have to reach them out via mail describing the issue and they will raise an incident and work on it, incase if you don't receive the incident number from InfoSys team in 30 minutes after you send the mail, you may reach them out over phone and ask the status and get a ticket created via phone if necessary.
While you send the mail to the InfoSys team, include the CGI BW team members or you can use DL for CGI BW Support Team i.e. $SBS IS Appli Techno Report Support-BW CGI < SBS-IS-Appli-Techno-Report- Support-BW.CGI@solvay.com > and concerned people from SOLVAY.
In case of high priority incident: Call the Service Desk
Toll-free numbers are available in Belgium 80072467, Brazil 8008913604, France 800992708, Germany 8001167050, India 8000404038, Portugal 800819261, Singapore 8008526740 and Thailand 1800014112
All other countries please use +44 207-715-5962 or +1 866-596-5680. Take reference to the written email, if applicable; ask for ticket creation, explain business impact and urgency
We can reach out below Members , who leading the Basis Team for solvay from Infosys in case of any escalation
- raghu.mudiganti-ext@solvay.com _ Lead
- Ramarao Vemuri <ramarao.vemuri-ext@solvay.com> - Lead
- Manoher Arvapalli <manoher.arvapalli@infosys.com> - Manger
Instances of system issues
- Job terminated in the source due to a memory issue.
- Short dumps at the source system (like TNEW_PAGE_ALLOCATION)
- .Job stuck in the source.
- Issues related to RFC and IDOCS
SDK_PCM Datasource Contact Details
Emergency Access Request (Fire-Fighter) Work Step By Step
W Iki Page : BW Firefighter
New URL for FFID Request - https://grcprd.eua.solvay.com:44375/sap/bc/webdynpro/sap/grac_uibb_end_user_login?sap-client=400#