computeMatrix using deepblue doesn't get data for beforeRegionStartLength/afterRegionStartLength

[jayoung]

Hi there,

I've recently started using deepTools - I love it. Thanks for all your work on this. I also started playing with using ENCODE data via deepBlue - also a really great feature.

I think I found a bug, and it's probably an easy one to fix. I should say upfront I'm using version 3.0.1 and have not tested 3.0.2. Apologies for that - I'm python-naive and am reliant on my sysadmins to install updates - but I don't see anything related in the changeLog, so thought I'd report it.

It seems like the bigWig file that computeMatrix obtains via deepBlue when I give it an Encode ID contains only the coverage data for the regions themselves but not the flanking regions I specified - it looks like the beforeRegionStartLength/afterRegionStartLength options didn't get applied before requesting the bigWig file - only later when it computes the matrix.

I started with an example like your mouse enhancers plot from the gallery - http://deeptools.readthedocs.io/en/latest/content/example_gallery.html#dnase-accessibility-at-enhancers-in-murine-es-cells. I wasn't sure of the right ENCODE id to use, so I'm probably using a different (but very similar) set of raw data.

first a sanity check - I get output very similar to yours if I manually download the Encode bigWig file

1. I download a bigwig file for Encode data ENCFF001OIK
   https://www.encodeproject.org/files/ENCFF001OIK/@@download/ENCFF001OIK.bigWig
   and I call it ENCFF001OIK.downloaded.bigWig

2. I download your example bed file Whyte_TypicalEnhancers_ESC.bed

3. I run computeMatrix+plotHeatmap and I get a very similar plot to yours:

computeMatrix reference-point -S ENCFF001OIK.downloaded.bigWig -R Whyte_TypicalEnhancers_ESC.bed --referencePoint center -a 2000 -b 2000 -out matrix_Enhancers_DNase_ESC.tab.gz --outFileNameMatrix matrix_Enhancers_DNase_ESC.tab.txt
plotHeatmap -m matrix_Enhancers_DNase_ESC.tab.gz -out matrix_Enhancers_DNase_ESC.heatmap1.png 

next I try using deepBlue to give me the same thing. I keep the temp bigWig file so I can take a look, as well as getting the matrix in a format I can look at:

computeMatrix reference-point -S ENCFF001OIK -R Whyte_TypicalEnhancers_ESC.bed --referencePoint center -a 2000 -b 2000 -out matrix_Enhancers_DNase_ESC_deepBlue.tab.gz --deepBlueTempDir /fh/scratch/delete30/malik_h --deepBlueKeepTemp --outFileNameMatrix matrix_Enhancers_DNase_ESC_deepBlue.tab.txt
plotHeatmap -m matrix_Enhancers_DNase_ESC_deepBlue.tab.gz -out matrix_Enhancers_DNase_ESC_deepBlue.heatmap1.png 

The plot shows coverage signal within the enhancer regions themselves, but in the flanking 2000bp it now shows missing data (swathes of black). I examine the bigWig file (or the text format matrix) and I see that they both only contain data within the regions, not in the flanks.

Does this reproduce with v 3.0.2? I'm guessing this will be a simple fix? Hope so!

thanks again,

Janet Young

---

Dr. Janet Young

Malik lab
http://research.fhcrc.org/malik/en.html

Division of Basic Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Avenue N., A2-025,
P.O. Box 19024, Seattle, WA 98109-1024, USA.

tel: (206) 667 4512
email: jayoung ...at... fredhutch.org

---

[dpryan79]

Wow, that's interesting. Don't bother checking that in 3.0.2, the same thing will happen there. I'm not sure yet if this is a bug on our end (quite possible) or from deepBlue (less likely, though I think they're moving the server so perhaps something got mucked up there). I'll have to have a look at this on Monday when I'm back in the office, until then I encourage you to download the files you need. BTW, thanks for the nicely reproducible example, it'll make tracking down the cause of this easier. Devon

…

-- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany <devon.ryan@dzne.de>

On Fri, May 4, 2018 at 2:04 AM, jayoung ***@***.***> wrote: Hi there, I've recently started using deepTools - I love it. Thanks for all your work on this. I also started playing with using ENCODE data via deepBlue - also a really great feature. I think I found a bug, and it's probably an easy one to fix. I should say upfront I'm using version 3.0.1 and have not tested 3.0.2. Apologies for that - I'm python-naive and am reliant on my sysadmins to install updates - but I don't see anything related in the changeLog, so thought I'd report it. It seems like the bigWig file that computeMatrix obtains via deepBlue when I give it an Encode ID contains only the coverage data for the regions themselves but not the flanking regions I specified - it looks like the beforeRegionStartLength/afterRegionStartLength options didn't get applied before requesting the bigWig file - only later when it computes the matrix. I started with an example like your mouse enhancers plot from the gallery - http://deeptools.readthedocs.io/en/latest/content/example_ gallery.html#dnase-accessibility-at-enhancers-in-murine-es-cells. I wasn't sure of the right ENCODE id to use, so I'm probably using a different (but very similar) set of raw data. first a sanity check - I get output very similar to yours if I manually download the Encode bigWig file 1. I download a bigwig file for Encode data ENCFF001OIK https://www.encodeproject.org/files/ENCFF001OIK/@@download/ ENCFF001OIK.bigWig and I call it ENCFF001OIK.downloaded.bigWig 2. I download your example bed file Whyte_TypicalEnhancers_ESC.bed 3. I run computeMatrix+plotHeatmap and I get a very similar plot to yours: computeMatrix reference-point -S ENCFF001OIK.downloaded.bigWig -R Whyte_TypicalEnhancers_ESC.bed --referencePoint center -a 2000 -b 2000 -out matrix_Enhancers_DNase_ESC.tab.gz --outFileNameMatrix matrix_Enhancers_DNase_ESC.tab.txt plotHeatmap -m matrix_Enhancers_DNase_ESC.tab.gz -out matrix_Enhancers_DNase_ESC.heatmap1.png [image: matrix_enhancers_dnase_esc heatmap1] <https://user-images.githubusercontent.com/2569397/39607967-00974452-4ef3-11e8-8f35-803bd97a943c.png> next I try using deepBlue to give me the same thing. I keep the temp bigWig file so I can take a look, as well as getting the matrix in a format I can look at: 1. computeMatrix reference-point -S ENCFF001OIK -R Whyte_TypicalEnhancers_ESC.bed --referencePoint center -a 2000 -b 2000 -out matrix_Enhancers_DNase_ESC_deepBlue.tab.gz --deepBlueTempDir /fh/scratch/delete30/malik_h --deepBlueKeepTemp --outFileNameMatrix matrix_Enhancers_DNase_ESC_deepBlue.tab.txt plotHeatmap -m matrix_Enhancers_DNase_ESC_deepBlue.tab.gz -out matrix_Enhancers_DNase_ESC_deepBlue.heatmap1.png [image: matrix_enhancers_dnase_esc_deepblue heatmap1] <https://user-images.githubusercontent.com/2569397/39608050-a722c616-4ef3-11e8-819b-548b7f37158a.png> The plot shows coverage signal within the enhancer regions themselves, but in the flanking 2000bp it now shows missing data (swathes of black). I examine the bigWig file (or the text format matrix) and I see that they both only contain data within the regions, not in the flanks. Does this reproduce with v 3.0.2? I'm guessing this will be a simple fix? Hope so! thanks again, Janet Young ------------------------------ Dr. Janet Young Malik lab http://research.fhcrc.org/malik/en.html Division of Basic Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Avenue N <https://maps.google.com/?q=1100+Fairview+Avenue+N&entry=gmail&source=g>., A2-025, P.O. Box 19024, Seattle, WA 98109-1024, USA. tel: (206) 667 4512 email: jayoung ...at... fredhutch.org ------------------------------ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#700>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGGkGDCabJRkcIZqvZjG9Zjqej1Q4PwUks5tu5skgaJpZM4Tx9I9> .

[jayoung]

Thanks Devon. Yes, I've been downloading files: that works fine. Hope it's easy to fix!

[dpryan79]

I made an embarrassingly simple mistake in the DeepBlue code that seems to be causing this. I'm testing now if there are any other issues and, if not, I'll merge the fix in and release version 3.1.0.

[dpryan79]

This is now finally fixed in the develop branch and will work properly in the 3.1.0 release. In addition to the silly typos that caused the main issue, I also had an off-by-one error in the code, so all of the values were shifted by one nucleotide.

[jayoung]

Thanks for fixing this - appreciate it.