Copy Raw Data Fastq Files from Server to Your Local Machine

 

This week, I worked with the Solve-RD platform (for rare diseases). My PhD works on utilizing whole-exome sequencing trios dataset related to epilepsy encephalopathy to gain some information on de-novo variants at FutureNeuro Centre (RCSI Dublin). Some of the cases are solved, and some trios are still remaining unsolved.

To start testing the Solve-RD software known as GPAP, I copied all my testing datasets from the server into my local machine. The aims are to upload all these sequences files into the GPAP platform. The files are in Fastq format.

The easier way to copy the file from server to local is using scp command. I use MobaXterm for this purpose as I am working with Windows. The idea is simple as below:

$ scp serveruser@serverip:serverpath localpath  

To copy the file into my laptop:

$ scp username@ipaddress:/path/to/your/folder/data/in/server/*.fastq.gz /path/to/your/folder/in/your/local

NOTE: You need to run the scp command from the local machine, not on the remote. You don't need the ssh at all.

How to Permanently Delete your Trash Files using Terminal Linux

Sometimes when you move your files to Trash, it is not permanently deleted from the system. In order to remove it completely, you can follow this step below. 

In this tutorial, I have access to the server to delete the files from my local PC.

# Navigate to server from your local PC. I only have normal user permission access to the server known as HPC3. 
 novopc4@novopc2:~$ ssh app_user@hpc3 

 #Put in your access password (xxxxx) 
app_user@hpc3's password: xxxxx 
Last login: Fri Aug 28 09:17:13 2020 from novopc3.local -bash: /opt/sge/novo/common/settings.sh: Permission denied

 # Next, you can access using "su root" to get permission for deleting files later 
app_user@hpc3:~$ su root 

 # Granted the password again (xxxxx) --ask root password from your admin Password: xxxxx 
 Note:  now u login as a root, only root can have permission to delete files in the server

# Access to your files thrash location, you need to "cd" the location
 root@hpc3:~# cd /export/home/hamidah/.local/share/Trash/files 

# list out the files in your Trash using "ls" command root@hpc3:/export/home/hamidah/.local/share/Trash/files# ls 

# remove permanently your selected Trash directory related to files, root@hpc3:/export/home/hamidah/.local/share/Trash/files# rm -r * 

# remove permanently your selected Trash directory related to expunged, root@hpc3:/export/home/hamidah/.local/share/Trash/expunged# rm -r *

#Please note that if your are accessing your server as a owner file, you may straight away delete your files using
sudo rm -rf /export/home/hamidah/.local/share/Trash/files/*

How to update R version inside the R docker for Debian distro

Today lesson about on how to update R version inside the R docker


When i ran a docker for R, the new package need the R version of 3.6.x
Mine was R version 3.4.x

                                                                      How ya?



We can follow this steps to quick upgrade the R version for the Debian distro:

# Enter your docker and remove the old version of R

root@1234 $ apt-get remove r-base-core

# Replace  with bionic which is compatible with your current system

root@1234 $ add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/'

# add the key

$ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9

# Do the update

$ apt-get update

# Install new r-base of R version

$ apt-get install r-base

# Check the new version

$ apt-cache policy r-base r-base: Installed: 3.6.3-1bionic Candidate: 3.6.3-1bionic Version table: *** 3.6.3-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 100 /var/lib/dpkg/status 3.6.2-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.6.1-3bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.6.1-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.6.0-2bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.6.0-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.5.3-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.5.2-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.5.1-2bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.5.1-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.5.0-1bionic 500 500 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages 3.4.4-1ubuntu1 500 500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

Hope you guys enjoy!

How to validate a bam file

Hello

I will cover on how to quick check our BAM file after mapping/alignment process.

In my case, I use BWA as for the mapping process of WGS data.

To validate our BAM file, first we need to install Picard tool, developed by Broad Institutes (https://broadinstitute.github.io/picard/). I will use Bioconda to install Picard

#quick check on bam file

$ conda create -n Picard python=3.7 $ conda activate Picard $ conda install -c bioconda picard $ picard -h $ ValidateSamFile -h #test one file $ picard ValidateSamFile I=/path/to/bam/file/SRR030257.bam MODE=SUMMARY #you can see warning or error on the terminal

Solution:

We must add Read Group information to our dataset using Picard utility before proceed with the next step.

Conda Not Found!

Conda Not Found!

Conda is open-source and quite popular among the Python/R package manager.

Sometimes we got an issue with Conda when we want to activate the Conda environment.

Here is an example of error :

$ CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.

And the solution :

$ source ~/miniconda3/etc/profile.d/conda.sh

$ conda activate yourenv

Source script will reads and executes commands from filename in the current shell environment

From Bash man page (info source) :

. filename [arguments] source filename [arguments] Read and execute commands from filename in the current shell environment and return the exit status of the last command exe‐ cuted from filename. If filename does not contain a slash, file names in PATH are used to find the directory containing file‐ name. The file searched for in PATH need not be executable. When bash is not in posix mode, the current directory is searched if no file is found in PATH. If the sourcepath option to the shopt builtin command is turned off, the PATH is not searched. If any arguments are supplied, they become the posi‐ tional parameters when filename is executed. Otherwise the positional parameters are unchanged. The return status is the status of the last command exited within the script (0 if no commands are executed), and false if filename is not found or cannot be read.

Reference:

https://github.com/conda/conda/issues/7980
https://superuser.com/questions/46139/what-does-source-do