Computing and storage

1. User guides

Several user guides are available:

1.1 New user at CC-IN2P3

Please refer to the Getting-started page of the CC-IN2P3 documentation.

1.2 Storage areas

Please refer to the storage-areas page of the CC-IN2P3 documentation.

The /sps area is a semi-permanent storage area. Its size is 360 To and its use can be followed from:

  • global usage;
  • per user usage. Les données qui n'ont pas été accédées depuis plus d'un an sont copiées sur bandes dans l'espace ATLASLOCALGROUPTAPE.

1.3 Starting with grid

Please refer to the ATLAS wiki page. Specific instructions for French users are the following:

  • French grid certificates are delivered from the following site and also this one;
  • when you join the ATLAS VO, it is necessary to ask for the 'Group Name' /atlas/fr.

1.4 Acces to LOCALGROUPDISK

As ATLAS France member, according that you have indeed added the 'Group Name' /atlas/fr during your registration to the ATLAS VO, you can write on the LOCALGROUPDISK areas of French sites, even if you are supposed to write only on the one of your own group:

  • GRIF-IRFU_LOCALGROUPDISK
  • GRIF-LAL_LOCALGROUPDISK
  • GRIF-LPNHE_LOCALGROUPDISK
  • IN2P3-CPPM_LOCALGROUPDISK
  • IN2P3-LAPP_LOCALGROUPDISK
  • IN2P3-LPC_LOCALGROUPDISK
  • IN2P3-LPSC_LOCALGROUPDISK

There exists a common area for the different groups at CC-IN2P3:

  • IN2P3-CC_LOCALGROUPDISK

These areas can be monitored from the following Grafana page.

1.5 Starting with Athena

There is no specific instructions on how to use Athena at CC-IN2P3. This is done through /cvmfs/atlas.cern.ch. Typically:

  export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
  source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
  lsetup "root 6.20.06-x86_64-centos7-gcc8-opt"
  alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'
  setupATLAS

1.6 User support

Please refer to the user-support page. In particular you can have a look at the user portal or submit a ticket to the helpdesk.

1.7 Using the computer farm

Please refer to the computing-introduction page of the CC-IN2P3 documentation.

2. Data management at CC-IN2P3

The Distributed Data Management group is in charge of the system to manage access to ATLAS data that is distributed at sites all around the world. The system consists of a bookkeeping system (dataset-based) and a set of local site services to handle data transfers, building upon Grid technologies. The software stack is called DQ2. These pages are dedicated to the use of DDM at CCIN2P3. Most of general information can be found in:

Files in the Grid can be referred to by different names: Grid Unique IDentifier (GUID), Logical File Name (LFN), Storage URL (SURL) and Transport URL (TURL). While the GUIDs and LFNs identify a file irrespective of its location, the SURLs and TURLs contain information about where a physical replica is located, and how it can be accessed.

2.1 DDM setup

With cvmfs, the same set of commands can be used at various sites, including CCIN2P3 and CERN:

  export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
  source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh
  lsetup rucio
  export DQ2_LOCAL_SITE_ID=IN2P3-CC_SCRATCHDISK

The last commands tells rucio where you run from to optimise data transferts. You need a grid proxy created by voms-proxy-init to use the DDM commands.

2.2 dCache au CC-IN2P3

dCache is a sophisticated system which is capable of managing the storage and exchange of hundreds of terabytes of data, transparently distributed among dozens of disk storage nodes or magnetic tape drives in hierarchical storage managers (hsms). It is jointly developed by DESY and Fermilab. A large dCache system has been deployed into CCIN2P3 production system for the benefits of ATLAS users at CCIN2P3. CCIN2P3 dCache system works as a disk caching system as a front end for Mass Storage System --- HPSS system.

2.3 Which data are located at the CCIN2P3

The grid space is divided into space tokens with specific access rights, e.g. DATADISK is for official ATLAS-wide dataset from production. The users can write on the SCRATHDISK (with a limited quota and lifetime of data), LOCALGROUPDISK from their cloud or specific performance/physics token with special privileges (e.g. PHYSHIGGS).

List the datasets available on the target disk space can be obtained by rucio:

 rucio list-datasets-rse IN2P3-CC_LOCALGROUPDISK

2.4 How to replicate data

The most general use case is that you have send your analysis on grid. Jobs have been send on a certain site and output saved there on a SCRATCHDISK with a limited quota and lifetime. It is thus necessary to replicate them on 'your' site. The replication can be requested at submission time with pathena/prun (destSE) or requested later:

  • 'freeze' your dataset;
  • use the tool Data Transfer Request Interface (R2D2) and follow instructions.

For archiving purpose, users can copy datasets to LOCALGROUPTAPE but please make sure there are no small files in the dataset (e.g. logfiles) as this does not work well with tapes. Transfers to LOCALGROUPTAPE are subject to approval by the cloud support.

2.5 How to access data at CC-IN2P3

Access to data located on a grid space is possible directly from CC-IN2P3 machines using the protocol XRootd.

The list of "files"/url from a dataset (container) can be obtained by:

 rucio list-file-replicas --rse IN2P3-CC_LOCALGROUPDISK --protocol root --domain lan mydataset

(Note the use of --domain to specify local access, as there are dedicated xrootd servers for local jobs and interactive use from CC-IN2P3 machines, the servers for external access are limited in the number of connections).

You can access directly files in your JO in Athena with xrootd :

 EventSelector.InputCollections = ["root://ccxrootdatlas.in2p3.fr:1094/pnfs/in2p3.fr/data/atlas/datafiles/csc11/recon/csc11.005200.T1_McAtNlo_Jimmy.recon.AOD.v11004103/csc11.005200.T1_McAtNlo_Jimmy.recon.AOD.v11004103._00111.pool.root.1"]

Direct access from ROOT can be done after setting up a recent version of ROOT:

 lsetup root
 TFile* xrfile = TFile::Open("root://ccxrootdatlas.in2p3.fr:1094/pnfs/in2p3.fr/data/atlas/atlaslocalgroupdisk/mc11_7TeV/NTUP_SMWZ/e861_s1310_s1300_r3043_r2993_p833/mc11_7TeV.145002.Pythia6_DYmumu_120M250.merge.NTUP_SMWZ.e861_s1310_s1300_r3043_r2993_p833_tid653896_00/NTUP_SMWZ.653896._000001.root.1")

Finally, you can copy locally a file on your interactive machine or inside your batch job:

 xrdcp root://ccxrootdatlas.in2p3.fr:1094/pnfs/in2p3.fr/data/atlas/atlaslocalgroupdisk/mc11_7TeV/NTUP_SMWZ/e861_s1310_s1300_r3043_r2993_p833/mc11_7TeV.145002.Pythia6_DYmumu_120M250.merge.NTUP_SMWZ.e861_s1310_s1300_r3043_r2993_p833_tid653896_00/NTUP_SMWZ.653896._000001.root.1 mylocal_file

If you think that a file is not accessible (lost or not), here is a list of points to check before contacting the helpdesk. Commands refer to the previous sections:

  • check the existence of the file : Use xrdcp to copy the file locally;
  • Check the accessibilty of the file with root: Open the file with root commands but check that you defined the same ROOT version as in your Athena program.

3. Grid sites monitoring

The monitoring of the grid sites is done by the ADC shifters, syst-admins and members of the 'cloud-FR support'. You can also look at the following information:

  • the monitoring of your own/atlas usage at CC-IN2P3 is available from the user portal;
  • monthly reports of the cloud-FR are available at the following page.

4. Cloud Computing

Please refer to the following pages:

5. High Performance Computing

For CC-IN2P3 and CERN, please refer to the following pages:

  • CC-IN2P3 has a HPC plateform as well as a GPU farm. See this page for details on these machines; *

For other sites in French universities:

At National level:

  • IDRIS (Institut du développement et des ressources en informatique scientifique), in particular the Jean Zay machine;
  • GENCI(Grand équipement national de calcul intensif).

At European level: * PRACE; * EuroHPC.

6. Quantum Computing

Please refer to the following pages:

  • at CNRS/IN2P3 : QC2I, Quantum computing for the two infinites
  • at Sorbonne Université : QQICS, Quantum Information Center Sorbonne
  • at Université Paris Saclay: QUANTUM, centre en sciences et technologies quantiques

7. Machine learning

Please refer to the following pages:

  • School Of Statistics 2021 (indico), Action Nationale de Formation CNRS/IN2P3, 18-20 January 2021;
  • Machine Learning pour informaticiens (indico), Action Nationale de Formation CNRS/IN2P3, September 2020;
  • IML group at CERN : (indico) workshop, September 2020.